JP5227841B2 - Ambient monitoring device - Google Patents

Description

  The present invention relates to a surrounding monitoring apparatus that monitors obstacles around a work machine, and particularly relates to risk presentation to a work machine operator and danger avoidance.

  Some work machines such as hydraulic excavators are equipped with a distance sensor for monitoring the surroundings in order to prevent a contact accident between the machine and surrounding workers. One of such devices is described in [Patent Document 1]. This device is equipped with a millimeter wave radar to measure obstacles around the machine, measures the distance to the obstacle, and the state of the obstacle (moving / stopping) within the operating position range of the work machine (front work machine) ) Restricts the operation of the machine.

In the work machine of [Patent Document 2], a camera that acquires image data is mounted on a work machine such as a hydraulic excavator, and the image data of the size and aspect ratio of the detected object is determined. It is determined whether or not there are workers.

JP 2002-47692 A JP 2007-23486 A

Three-dimensional image measurement, pp.92-99, Shosendo, 1992, published by Seiji Iguchi and Kosuke Sato

  However, these conventional techniques have the following problems.

  The device disclosed in [Patent Document 1] is equipped with a millimeter wave radar, measures the distance to the obstacle, and depends on the state of the obstacle (moving / stopping) within the working machine (front working machine) operating position range. Although the operation of the machine is limited, since all the three-dimensional objects existing in the operation position range are detected, unnecessary operation of the machine is restricted, resulting in a reduction in work efficiency.

Furthermore, in the apparatus disclosed in [Patent Document 2], the presence of a person around the machine is determined by determining the image data of the size and aspect ratio of the detected object using the image data detected by the image sensor. However, in this technique, when a person exists in the vicinity of the machine, it is difficult to determine whether the person exists correctly using image data.

The object of the present invention is to solve these problems, and in the monitoring range around the entire work machine, for the driver who is working, regardless of the distance between the work machine and the obstacle, It is possible to accurately determine the state of the object and the state, inform the operator of the danger level information for surrounding scenes and obstacles before changing the operation of the machine, and perform machine operations to avoid the minimum necessary danger. An object of the present invention is to provide a surrounding monitoring device for a work machine that can ensure safety and improve work efficiency.

To achieve the above object, the present invention provides an apparatus for monitoring an obstacle around a work machine, an image sensor for detecting image data of the obstacle, an electromagnetic wave sensor for detecting position data of the obstacle, Obstacle detection means for detecting the presence of an obstacle such as an operator or an object around the work machine by an image sensor or an electromagnetic wave sensor , and the obstacle and the work machine for calculating the positional relationship between the detected obstacle and the work machine If the positional relationship between the detected obstacle and the work machine is within a distance of a predetermined value, whether or not the obstacle corresponds to a person depending on the presence or absence of a human head feature in the image data a feature calculating means for determining a mechanical state capturing means of the posture change of the turning and movement of the working machine from the positional relationship of the calculated, the machine operating state capture means the turning or moving state of the machine If forme Ri, the obstacle is determined persons, or the contact risk calculation means for calculating the contact risk level corresponding to an object other than a person, the alarm level suitable for risk of the contact risk calculation means An alarm level determining means and an alarm output means for outputting alarm contents to the output device from the determined alarm level are provided.

Furthermore, the surroundings monitoring device of the present invention is characterized in that, when the positional relationship between the obstacle detected by the feature calculation unit and the work machine is equal to or greater than a predetermined distance, the obstacle is calculated based on the aspect ratio of the obstacle in the image data. It is characterized in that it is determined whether or not corresponds to a person.

Furthermore, in the surroundings monitoring device of the present invention, when the positional relationship between the detected obstacle and the work machine is greater than or equal to a predetermined value distance, the feature calculation means sets the obstacle extraction region in the image data as the head, It is characterized by determining whether or not the obstacle corresponds to a person by dividing the body part and the lower part and calculating respective contour shapes.

Furthermore, in the surroundings monitoring device of the present invention, when the positional relationship between the obstacle detected by the feature calculation unit and the work machine is within a predetermined distance, a circular contour is used as a human head feature in the image data. It is characterized by detecting the presence or absence of.

In order to achieve the above object, according to the present invention, there is provided a peripheral monitoring device for monitoring an obstacle around a work machine, an image sensor for detecting image data of the obstacle, and an electromagnetic wave for detecting position data of the obstacle. Obstacle detection means for detecting the presence of an obstacle such as a worker or an object around the work machine by the sensor, the image sensor or the electromagnetic wave sensor , and the position of the obstacle and the machine from the machine body of the detected obstacle If the positional relationship between the obstacle and the machine for calculating the relationship and the detected obstacle and the work machine is within a predetermined distance, the obstacle is determined depending on the presence or absence of a human head feature in the image data. and determining characteristics calculating means whether an object corresponds to a person, and machine state capturing means of the posture change of the turning and movement of the working machine from the positional relationship of the calculated, the operating state capturing means of the machine When incorporating a turning or moving state of OMRON will the contact risk calculation means and the obstacle to calculate a person, or the contact risk level corresponding to an object other than a person, suitable for the risk of the contact risk calculation means An alarm level determining means for determining the alarm level, and an alarm output means for outputting the alarm content from the determined alarm level to the monitor, and the obstacle displays differently on the monitor depending on a person or a non-person. It is characterized by this.

  Furthermore, the surroundings monitoring device of the present invention is characterized by displaying a positional relationship between the range of turning and movement and the obstacle.

  Furthermore, the surroundings monitoring device of the present invention is characterized in that the alarm output means displays differently on the obstacle according to the state of the turning or movement posture change of the work machine.

The surroundings monitoring device according to the present invention provides an obstacle within a working machine (front working machine) operating position range to a driver who is working regardless of the distance between the working machine and an obstacle in the monitoring range around the working machine. In order to present the danger level of an object (a worker or an object other than a worker ) to the operator, the type of machine and obstacle, that is, whether there is a worker or an object exists when a danger occurs Since the grasp becomes obvious and the appropriateness judgment of the machine operation can be confirmed accurately and instantaneously, the operator can accurately and effectively grasp the contact risk level and the position and direction of the obstacle and the worker without reducing the work efficiency. Thus, obstacles can be avoided, safety can be ensured, and work efficiency can be improved.

In addition, the surroundings monitoring device according to the present invention is provided with an alarm output means for outputting alarm contents from the determined alarm level to the monitor for a driver who is working in the monitoring range around the entire working machine. It is possible to ensure safety and improve work efficiency by accurately displaying different indications on the monitor depending on whether the obstacle is a person or a non-person regardless of the distance to the object.

It is a figure which shows the external appearance of the hydraulic shovel which is an example of a working machine. It is a figure which shows the example of a procedure of the operation | movement of an excavator work which is an example of a working machine, and the flow of a process. It is a figure which shows the whole structure of the periphery monitoring apparatus for working machines using this invention. It is a figure which shows the whole structure in another form of the periphery monitoring apparatus for working machines using this invention. It is explanatory drawing which shows the example of an outline of the positional relationship of the obstruction and the working machine in the circumference | surroundings of a working machine. It is a figure which shows the example of an outline of the detection range of a camera and a millimeter wave radar in the detection apparatus of an obstruction detection means. It is explanatory drawing which shows the example inside an obstruction detection means. It is a figure which shows the example of an outline of the working range around the machine of a camera imaging | photography scene, and a machine. It is explanatory drawing which shows the other example in a change area extraction part. It is explanatory drawing which shows the imaging | photography scene example of a camera. It is a figure which shows the example of expansion imaging | photography of the detection rectangular frame of a millimeter wave radar. It is a figure which shows the example of a procedure of a feature calculation part. It is a figure which shows the internal example of a machine state taking-in means. It is a figure which shows the other example of a determination that a vehicle body reverses. It is a figure which shows the example of an outline of the positional relationship calculation means of an obstruction and a machine. It is a figure which shows the example of a procedure of the positional relationship calculation means of an obstruction and a machine. It is a figure which shows the example of a procedure inside a contact risk degree calculation means. It is a figure which shows the example of a procedure inside an alarm level determination means. It is a figure which shows the example of a procedure inside an alarm output means. It is a figure which shows the example which superimposes a detection mark on the obstruction of a camera image. It is a figure which shows the example of a person warning alarm and a warning warning mark to a person. It is a figure of the example of a contact mark and a machine operation stop signal mark to a person contact warning and a contact person. It is explanatory drawing which shows the example of a procedure of an apparatus operation state judgment means. It is explanatory drawing which shows the example of a procedure of an apparatus control means. It is a figure which shows the example of a procedure of an apparatus control signal output means.

  An example of a work machine periphery monitoring apparatus using the present invention will be described with reference to the drawings.

  FIG. 1 shows an overview of a hydraulic excavator that is an example of a work machine. The hydraulic excavator includes an articulated front working machine 1A including a boom 1a, an arm 1b, and a bucket 1c that rotate in a vertical direction, and a vehicle body 1B including an upper swing body 1d and a lower traveling body 1e. The upper swing body 1d is provided with an operator cab 1f. The base end of the boom 1a of the front work machine 1A is supported by the front part of the upper swing body 1d. The boom 1a, arm 1b, bucket 1c, upper swing body 1d and lower traveling body 1e are respectively a boom cylinder 3a, arm cylinder 3b, bucket cylinder 3c, swing motor 3d (not shown in FIG. 1) and left and right travel motors 3e, It is driven by each actuator 3f (not shown). In addition, the boom 1a, the arm 1b, the bucket 1c, and the upper swing body 1d include angle detectors 8a, 8b, 8c, and 8d that detect the respective rotation angles. Further, a camera 13a is installed behind the machine to capture a rear scene, and a millimeter wave radar (distance sensor) 14a is installed to measure the distance from the machine rear to the obstacle. A camera 13b is installed on the right side of the machine to photograph a right side scene, and a distance sensor 14b (not shown) is installed to measure the distance from the right side of the machine to the obstacle.

  FIG. 2 is an explanatory diagram illustrating an example of a procedure of operations and processing flows in the operation of a hydraulic excavator that is an example of a work machine. 2 is an example in which a camera 13a and a millimeter wave radar 14a are used as distance sensors.

  At a certain time 1701, the camera 13 a captures a scene, and when the machine starts excavation 1702, image processing 1706 is performed using the scene captured by the obstacle detection unit 600 to detect whether there is an obstacle 1703 during excavation work. . On the other hand, the millimeter wave radar 14a always performs sensing 1712 while the machine is working. Therefore, during excavation operation 1703, the presence / absence of the obstacle, the type of obstacle (person or other than the person), and the distance data of the obstacle of the millimeter wave radar 14a are integrated (fusion). The result of the presence / absence of obstacle, the type and distance data to the obstacle is output 1707. The operator confirms this result with a monitor and voice, and operates the machine while confirming the degree of danger around the change direction when changing the operation (backward movement, turning, etc.).

  Now, when a person is present in the right side operation position range and the front work machine is turned 1704 to the right (confirmed by the signal 1708 of the front work machine 1A and the optical flow 1709 using the camera image), Since there is a high possibility of contact, the front work machine is stopped as control 1710 for avoiding contact to ensure the safety of the person. Further, when an object exists in the right-side movement position range, the front work machine 1A is slowly operated as a control 1710 for avoiding contact. At this time, after the contact avoidance control, the obstacle detection means 600 using the photographing scene of the camera 13a may be stopped and resumed at the start of excavation.

  On the other hand, when a person is present in the backward movement range and the lower traveling body moves backward 1705, there is a high possibility that the lower traveling body will come in contact with the person, so the movement of the lower traveling body is stopped as control 1711 to avoid contact, Ensuring the safety of people.

  Here, the detection by the camera 13a is shown as an example only during excavation, but sensing may be performed constantly during the operation of the machine.

  An embodiment of the present invention is shown in FIG. In FIG. 3, an image processing apparatus 90 includes an obstacle detection means 600, an obstacle-machine positional relationship calculation means 1100, a contact risk calculation means 1200, an alarm level determination means 1300, an alarm output means 1400, and a machine state capturing means 700. Consists of.

  In the processing in the image processing apparatus 90, when the camera 13a captures a rear scene of the machine, the obstacle detection unit 600 performs image processing using the captured scene of the camera 13a, and obstacles such as workers and objects around the work machine. Detect the presence or absence of. When there is an obstacle in the obstacle detection means 600, the positional relationship calculation means 1100 between the obstacle and the machine calculates the positional relationship between the distance from the machine body to the obstacle and the machine and the direction, and the contact risk calculation means 1200. However, the obstacle contact risk level is determined based on the positional relationship calculated by the obstacle-to-machine positional relationship calculating unit 1100 and the operations such as excavation, turning, and movement of the work machine captured by the machine state capturing unit 700. calculate. After that, when the alarm level determining unit 1300 determines an alarm level from the obstacle contact risk level calculated by the contact risk calculating unit 1200 and outputs the alarm level to the alarm output unit 1400, the alarm output unit 1400 outputs an alarm corresponding to the level. The contents are output to an output device 1500 such as a monitor or voice. The operator determines the operation of the machine by grasping the result of the output device 1500 visually or by voice.

  Here, the image processing apparatus 90 may be configured by a PC that can realize image processing or an image processing dedicated apparatus. The output device 1500 may be provided in the driver's cab, and may be anywhere as long as the operator can visually check it.

  Another embodiment of the present invention is shown in FIG. In FIG. 4, an image processing apparatus 90 includes an obstacle detection unit 600, an obstacle-to-machine positional relationship calculation unit 1100, a contact risk calculation unit 1200, an alarm level determination unit 1300, an alarm output unit 1400, and a machine state capturing unit 700. , Equipment operation state determination means 1600, contact avoidance control means 1700, equipment control signal output means 1800. In the processing in the image processing apparatus 90, when the camera 13a captures a rear scene of the machine, the obstacle detection unit 600 performs image processing using the captured scene of the camera 13a, and obstacles such as workers and objects around the work machine. Detect the presence or absence of. When there is an obstacle in the obstacle detection means 600, the positional relationship calculation means 1100 between the obstacle and the machine calculates the positional relationship between the distance from the machine body to the obstacle and the machine and the direction, and the contact risk calculation means 1200. However, the obstacle contact risk level is determined based on the positional relationship calculated by the obstacle-to-machine positional relationship calculating unit 1100 and the operations such as excavation, turning, and movement of the work machine captured by the machine state capturing unit 700. calculate.

  Further, when the alarm level determining unit 1300 determines an alarm level from the obstacle contact risk level calculated by the contact risk calculating unit 1200 and outputs the alarm level to the alarm output unit 1400, the alarm output unit 1400 outputs an alarm corresponding to the level. The contents are output to an output device 1500 such as a monitor or voice.

  The operator determines the operation of the machine by grasping the result of the output device 1500 visually or by voice, but if the operator is distracted by other operations and overlooks the output device 1500, the device operation state Judgment means 1600 judges the posture change such as excavation, turning and movement from the contact risk calculated by the contact risk calculation means 1200 and the state of the work machine captured by the machine state capturing means 700, and further determines the alarm level. When the operation state of the device is determined from the alarm level determined by the means 1300, the contact avoidance control unit 1700 controls the device operation according to the determination result of the device operation state determination unit 1600 to reduce the speed during reverse or turning. A control signal that stops or avoids contact is output. Then, the equipment control signal output means 1800 transmits the output of the contact avoidance control means 1700 to the equipment and controls the operation of the work machine. As a result, contact with a person or the like can be avoided and safety can be ensured, and further, machine operation can be performed with the minimum necessary, and work efficiency can be improved.

  Here, the image processing apparatus 90 may be configured by a PC that can realize image processing or an image processing dedicated apparatus.

FIG. 5 is an explanatory diagram showing an example of an overview of the positional relationship between an obstacle (a person and an object) and the work machine around the work machine. In FIG. 5, the camera 13 and the millimeter wave radar 14a are installed in the rear and the camera 13b and the millimeter wave radar 14b (not shown) are installed on the right side in monitoring the surroundings of the work machine. 15a, 15b, 15c, an object sandbag 16, a color cone 17 and a pole side cone 19 are present, and a worker 15d is present on the right side.
The monitoring range is the work range of the front work machine 1A, but when the machine is excavating, the rear work range 18a and the right work range 18b are set as the longest range of the front work machine 1A.

  Now, when the machine is excavating, the alarm level determination means 1300 is dangerous because the person 15a is closest to the machine, and makes the alarm level the highest, and then increases the alarm level of the persons 15b and 15d. Since the sandbag 16, the color cone 17, the pole side cone 19 and the like are objects, the alarm level is lowered.

  FIG. 6 is an explanatory diagram showing an example of an overview of the detection range when the camera 13a is used as the detection device in the obstacle detection means 600 and when the millimeter wave radar is used. In FIG. 6, although it is an example of a hydraulic excavator, the height of the lower traveling body 1e is 1.0 m or more, and the installation height of the millimeter wave radar 14a has to be installed at a position higher than 1.0 m. Now, since the millimeter wave radar 14a can perform the horizontal scanning 23 and the vertical direction is difficult, the person 15a (height of about 0.8 m) sitting directly below the millimeter wave radar 14a is It becomes a blind spot and becomes undetectable. Therefore, if the camera 13a is installed at the top of the vehicle body 1B so that the camera 13a is photographed directly under the camera 13a, it is possible to shoot 20a in the horizontal and vertical directions, and the person 15a can also be photographed. do not do. Therefore, by using the camera 13a, the detection performance is improved and the safety is also increased.

  FIG. 7 is an explanatory diagram showing an example of the inside of the obstacle detection means 600. The image input unit 200 inputs a shooting scene of the camera 13a, and the background image creation unit 210 creates a reference background image (a previous image, an input image before n frames, a normal image automatically generated, or the like). The vibration correction unit 220 compares the background image of the background image creation unit 210 with the image of the image input unit 200, and determines that there is vibration when the difference for each pixel is greater than or equal to a threshold value in most screens. The amount of movement is calculated and corrected by, for example. After correcting the shift between images by the vibration correction unit 220, the difference image creation unit 230 for each pixel creates a difference image for each pixel. The change area extraction unit 240 performs binarization processing in which the difference image created by the difference image creation unit 230 for each pixel is set to 0 when the difference image is less than a predetermined threshold (about 7 to 15) and 1 or more when the difference image is greater than that. To extract the change area of the obstacle. Whether or not the change area has been extracted is determined by determining whether or not the change area extraction unit 240 has an area whose area is greater than or equal to a predetermined threshold (large near the camera and small in the distance, about 50 to 200). Is extracted as an obstacle area, and if it is not extracted, no obstacle area 260 is set as no obstacle area.

  On the other hand, if the determination 250 of whether or not the change area has been extracted extracts the area of the obstacle, the feature calculation unit 270 uses the aspect ratio as the feature of the obstacle, and the extraction area is displayed in the head, body, and lower part. Dividing into three parts, the respective contour shapes and the like are calculated. The determination 280 of whether or not a person is a feature is that the aspect ratio of the feature amount calculated by the feature calculation unit 270 is vertically long, the head of the extraction region has a fan-shaped contour, and the shoulder portion of the body portion has an oblique contour. The torso has a vertical contour, and features such as an inverted V-shaped contour and vertical contour at the bottom. If there are a plurality of these features, it is a person. judge. When it is determined that there are a plurality of person features in the determination 280 as to whether or not there is a person feature, it is determined that there is a person 300, and when there are only one or less person features, it is determined that there is an obstacle 290 other than a person.

  FIG. 8 is an explanatory diagram showing an example of an overview of a machine and a work range around the machine in a shooting scene of the camera 13a. When the millimeter wave radar 14a is installed at a height of 1.0 m at the bottom of the rear surface of the vehicle body 1B, and the camera 13a is installed at the top of the vehicle body 1B at a depression angle so that the camera 13a is photographed directly below, The shooting scene 13a is a shooting range 50a from directly below the camera, and the millimeter wave radar 14a has an object less than 1.0 m in the dead zone (away from the millimeter wave radar 14a) in the range 50c depending on the installation from directly below its own millimeter wave radar 14a. The height becomes lower). Further, the work range of the front work machine 1A is the longest work range 50b of the front work machine 1A during excavation work, and the range 50d depending on the turning posture of the front work machine 1A during turning. Therefore, the out-of-work area 51a of the front work machine 1A in the imaging range 50a is excluded from the detection target area of the obstacle detection unit 600. On the other hand, in the shooting range 50a, the area from the work range 50b or 50d directly below the camera is the image processing area 51b, the area from the camera right to the range 50c is the radar blind spot area 51c, and the area from the range 50c to the work range 50b or 50d is the camera. Since it can be detected by both 13a and the millimeter wave radar 14a, it is set as a fusion region 51d. As a result, since the surroundings are excluded from the detection target area outside the work range and no blind spots are generated, the reliability and the safety around the machine can be improved.

  FIG. 9 is an explanatory diagram illustrating another example of the change area extraction unit 240. In the fusion region 51d, the distance and direction to the three-dimensional object by the millimeter wave radar 14a are detected. Position 41, position 42, position 43, and position 44 from the side closer to the millimeter wave radar 14a are excluded because they are outside the work range 51a.

Here, conversion between the ground 3D coordinate system and the camera screen 2D coordinate system is possible, and [Non-Patent Document 1] describes a technique for a more general pinhole camera model. The outline is converted from the ground 3D coordinate system to the camera 3D coordinate system and then converted to the camera screen 2D coordinate system. That is, the camera 3D coordinate system is obtained by multiplying the combination of the ground 3D coordinate system and the camera position parameter by the side view angle parameter, the rotation angle parameter, and the rotation matrix related to the rotation angle parameter. Next, the camera screen 2D coordinate system is obtained by combining the camera 3D coordinate system and the scale parameter.

  Accordingly, a rectangular frame 41a assuming a solid object equivalent to 2 m from the position 41 with respect to the shooting scene 50a of the camera 13a, and a rectangular frame 42a assuming a solid object corresponding to 2 m from the position 42 are equivalent to 2 m from the position 43. A rectangular frame 43a assuming a three-dimensional object is calculated, and the rectangular frame 41a, the rectangular frame 42a, and the internal area of the rectangular frame 43a serve as an obstacle area extracting unit 240. Thereby, since the processing area in the photographing scene 50a of the camera 13a is limited, the detection performance can be improved and the processing speed can be improved.

  Further, when a mark is superimposed on the shooting scene 50a of the camera 13a from the distances and directions of the position 41, the position 42, and the position 43, a mark (for the convenience of description, the detection mark 41 and the mark below the object 15b). Is marked below the obstacle (object) 21 with respect to the position 42 (referred to as a detection mark 42 for convenience of explanation) and marked below the color cone (object) 17 with respect to the position 43. For this reason, the detection mark 43 is used. When this mark is superimposed on the shooting scene 50a, a detection mark is attached if a three-dimensional object exists. When this marking is output conspicuously on an output device 1501 such as a monitor, it can be used to assist the operator 1503 in determining the obstacle type, and work efficiency can be improved.

FIG. 10 is an explanatory diagram illustrating an example of a shooting scene of the camera 13a. As an example of a hydraulic excavator, the height of the lower traveling body 1e is 1.0 m or more, and the camera 13a is photographed in the uppermost part (about 2.0m) of the vehicle body 1B and directly under the camera 13a. If it is installed at a depression angle, an object near the camera 13a can be photographed, so no blind spot is generated. However, when a person is present near the camera 13a, only the head 601 is photographed at a position immediately below the camera 13a.
Further, in the vicinity of the camera 13a (around 1.0 m), the person 602 becomes an odd person with a large head and extremely small torso and legs. On the other hand, when it is away from the camera 13a (about 1.5 m or more), it becomes a human-like appearance 603 or 604. Therefore, in the feature calculation unit 270, the feature of the object is calculated separately in the vicinity region 610 and the far region 620 near the camera in the shooting scene of the camera 13a. That is, in the far region 620, the aspect ratio, the extracted region is divided into three parts, the head portion, the body portion, and the lower portion, and the respective contour shapes and the like are calculated. In the neighborhood area 610 near the camera, a determination is made as to whether the person is a person or not based on the presence or absence of a circular outline and a fan-shaped outline that are head features of the extraction area.

  On the other hand, the difference image creation unit 230, the change region extraction unit 240, the determination 250 of whether or not the change region has been extracted, and the feature calculation unit 270 for each pixel in the neighboring region 610 and the far region 620 may be different from each other. Good. That is, different image processing methods may be applied.

  As a result, it is possible to identify whether the detected object is a person or a person other than the person, and a detailed alarm level is determined according to the presence or absence of the person, thereby improving the safety of the worker (person).

FIG. 11 is an explanatory diagram showing an example of enlarged photographing of the detection rectangular frame 41a of the millimeter wave radar 14a.
Although the rectangular frame 41a is generated from the detection data of the millimeter wave radar 14a, the object feature calculation unit 270 preferably has an image with clear object features. Therefore, when shooting with the camera 13a and aiming at the rectangular frame 41a for enlarged shooting, a zoom image 48a is obtained. Since the object feature calculation unit 270 determines the feature of the object from the zoom image 48a, the determination 280 as to whether or not the object is a feature of the person can be performed with higher accuracy.

  In addition, the superimposed image may be displayed on the zoom image 48a in the descending order of the alarm level outside the work range 51a of the shooting scene 50a of the camera 13a.

FIG. 12 is an explanatory diagram illustrating a procedure example of the feature calculation unit 270. In step 271, it is determined whether or not the entire scene is a camera installation in which the entire body of the obstacle is photographed. If the camera is not installed in which the entire body is photographed, step 272 determines whether or not the extraction unit is an area near the camera. To do. If step 272 determines that the area is near the camera, step 273 sets a neighboring area 610 that is an area near the camera. Step 275 calculates the feature where the extraction region is an obstacle part, that is, calculates the feature such as the presence or absence of a circular contour that is the head feature of the extraction region and the fan-shaped contour, and step 280 If there is a circular outline and a fan-shaped outline, it is determined as a person, and the other is determined as an object other than a person.

FIG. 13 is an explanatory diagram showing an internal example of the machine state capturing means 700. First, step 701 takes in the output θ1 of the angle detector 8a that detects the rotation angle of the boom 1a, and step 705 uses it as a parameter for calculating the tip coordinates. Next, step 702 takes in the output θ2 of the angle detector 8b that detects the rotation angle of the arm 1b, and step 705 serves as a parameter for calculating the tip coordinates. Subsequently, Step 703 takes in the output θ3 of the angle detector 8c for detecting the rotation angle of the bucket 1c, and Step 705 serves as a parameter for calculating the tip coordinates.
Finally, Step 704 takes in the output θ4 of the angle detector 8d that detects the rotation angle of the upper swing body 1d, and Step 705 uses it as a parameter for calculating the tip coordinates. In step 705, when the tip coordinates of the bucket 1c are calculated from θ1, θ2, θ3, and θ4, step 706 determines the size (length) of the front work machine 1A and the turning direction of the upper swing body 1d from the lower traveling body 1e. calculate. Thereby, since the work ranges 50b and 50d can be calculated in conjunction with the front work machine 1A, it is possible to monitor the surroundings in the work range with high work efficiency.

  On the other hand, if it is determined in step 271 that the camera is installed so that the whole body is photographed, in step 276 the extracted area is the feature of the obstacle, that is, the aspect ratio, that is, the extracted area is the head, body, Dividing into three parts at the bottom, the respective contour shapes and the like are calculated. Further, when step 272 determines that the extraction unit is not an area near the camera, step 274 sets a far area 620 that is an area far from the camera, and the process of step 276 is performed.

  FIG. 14 is an explanatory diagram illustrating another determination example in which the vehicle body 1B moves backward. When the shooting scene of the camera 13a is used, a plurality of observation points 61 and 62 are set in advance. Now, the pattern 61a in which the pattern of the peripheral region of the observation point 61 at the time t 65a is most similar to the vicinity of the observation point 61 at the time t + 1 time 65 is searched and the amount of deviation is calculated. Similarly, the pattern 62a in which the pattern of the peripheral region of the observation point 62 at the time t 65a is most similar to the periphery of the observation point 62 at the time t + 1 is searched for the amount of deviation.

  Further, the pattern 62a in which the pattern of the peripheral region of the observation point 62 at the time t 65a is most similar to the periphery of the observation point 62 at the time t + 1 65 is searched for and the amount of deviation is calculated. If this is repeated in the same manner for the observation point, flows 61c, 62c, and 63c indicating the directions in which the search has been performed are obtained. As a whole, the flow becomes 65c (optical flow). The maximum number in the same direction of this flow indicates the moving direction, and since it is the downward direction, it can be seen that it is backward. If the direction of the optical flow is the right direction, it is a right turn.

  FIG. 15 is an explanatory diagram illustrating an example of an overview of the obstacle-to-machine positional relationship calculating unit 1100. When a person area 41a and a person area 41b and object areas 42a and 43a are detected during excavation 1703 with respect to a shooting scene of the camera 13a installed behind the center 1120 of the vehicle body 1B, the person area 41b is located on the vehicle body 1B. The person area 41a is next closest. In the object area 42a and the object area 43a, the object area 42a is close to the vehicle body 1B. Therefore, the person in the person area 41b has the highest contact risk.

  On the other hand, when the person area 41d and the person area 41e are detected during the excavation 1703 with respect to the shooting scene of the camera 13b installed on the right side of the center 1120 of the vehicle body 1B, the person area 41d is closest to the vehicle body 1B. The person area 41e is close. Here, when the vehicle body 1B turns 53 to the right, there is a risk that the person in the person area 41e will first contact the front work machine 1A. Therefore, the person in the person area 41e has the highest contact risk.

  FIG. 16 is an explanatory diagram illustrating an example of the procedure of the obstacle-to-machine positional relationship calculating unit 1100. Step 1101 determines the presence or absence of an obstacle. If step 1101 determines the presence or absence of an obstacle, step 1102 calculates the distance and direction from the center 1120 of the vehicle body 1B to the obstacle on the right side of the machine. 1103 calculates the distance and direction from the center 1120 of the vehicle body 1B to the obstacle behind the machine. From these, step 1104 calculates the proximity coefficient (value k that is calculated larger as the distance is closer) between the person in the area closest to the vehicle body 1B and the obstacles other than the person from the distance and direction.

  On the other hand, if it is determined in step 1101 that there is no obstacle, step 1105 sets the proximity coefficient between the person and the obstacles other than the person to k = 0.

FIG. 17 is an explanatory diagram showing an example of an internal procedure of the contact risk calculation means 1200. Step 1201 determines whether or not there is an obstacle. If there is no obstacle, that is, if the proximity coefficient k = 0 calculated by the positional relationship calculation means 1100, step 1213 determines that the contact risk coefficient α = 0.
If it is determined in step 1201 that there is an obstacle, step 1202 determines the previous operation of the machine. If it is determined that excavation is performed in the previous operation determination of the machine in step 1202, step 12015 determines that the risk factor β = 1 and performs step 1203. If it is determined in the step 1202 that the machine has been moved backward and turned in the previous machine operation determination, the risk coefficient β = 2 is determined in the step 1216 and the step 1203 is performed. Step 1203 determines the current machine operation. If it is determined in Step 1203 that the current machine operation is excavation, Step 1213 determines whether the person is a person or not. The risk factor α is determined to be 100, and if it is not a person, step 1214 determines that the contact risk factor α is 1.

  If it is determined in step 1203 that the current machine operation is backward, step 1210 determines whether the person is a person or the other. In the case of a person, step 1211 determines that the contact risk coefficient α = 400 × β. In the case other than a person, step 1212 determines that the contact risk coefficient α = 3 × β.

  Further, if it is determined in step 1203 that the current machine operation is a right turn, step 1204 determines whether or not it is within the turning trajectory or turning height range of the tip taken from the tip coordinates of the bucket 1c of the turning body. When it is outside the turning locus range, step 1207 determines that the contact risk coefficient α = 0, and when it is within the turning locus range, step 1205 is executed. Step 1205 determines whether or not the person is a person. In the case of a person, step 1206 determines that the contact risk coefficient α = 400 × β, and in the case of a person other than the person, step 1208 determines that the contact risk coefficient α = 3 × β. To do.

  FIG. 18 is an explanatory diagram illustrating an example of an internal procedure of the alarm level determination unit 1300. Step 1301 calculates the maximum turning trajectory range (Max) of the tip that does not depend on the machine state taken in from the turning body (front device), that is, when the front work machine 1A is extended to the maximum, and step 1302 calculates the alarm coefficient M. Is calculated (M = k × Max), and Step 1303 is performed. In step 1303, the proximity coefficient k to the obstacle (calculated by the positional relationship calculation means 1100 between the obstacle and the machine), the alarm coefficient M (calculated in step 1302), and the contact risk value α (contact risk calculation means 1200). Alarm value A (A = α × k × M).

  Step 1304 checks the range of step 1303 alarm value A. If A = 0, step 1305 is alarm level = 0, and if 0 <A ≦ 50, step 1306 is alarm level = 1 and 50 <A <100. If step 1307 is alarm level = 2, 100 ≦ A <200, step 1308 is alarm level = 3, if 200 ≦ A <300, step 1309 is alarm level = 4, and if 300 ≦ A, step 1310 is The alarm level is determined to be 5.

  FIG. 19 is an explanatory diagram showing an example of an internal procedure of the alarm output unit 1400. In step 1401, a detection mark is superimposed on an obstacle in the camera image, and in step 1402, the alarm level determined by the alarm level determination unit 1300 is checked. If the alarm level in step 1402 is 0, step 1403 is a process that does not display an alarm mark. If the alarm level is 1, step 1404 displays an obstacle warning display and a warning mark or sound for the obstacle to the output device 1500. When the alarm level is 2, step 1405 displays an obstacle alarm display and an alarm mark or sound for the obstacle to the output device 1500. When the alarm level is 3, step 1406 displays a person caution display and a caution mark for the person. Audio is output to the output device 1500. When the alarm level is 4, step 1407 outputs a person contact and a contact mark or audio to the person to the output device 1500. When the alarm level is 5, step 1408 displays a person contact. A mark that outputs a contact mark or sound to the output device 1500 and stops the machine operation such as reverse or turning. To display.

  As a result, the operator 1503 can grasp the degree of danger to obstacles existing in the working range of the machine from the screen of the video output device (monitor) 1501 and the sound of the audio output device (speaker) 1502, and there is a risk of contact. Since the reverse or turning of the machine is also stopped only when it is large, the operator 1503 can easily check the surrounding obstacle situation and can also ensure the safety of the person and the work efficiency.

  FIG. 20 is an explanatory diagram illustrating an example in which a detection mark is superimposed 1401 on an obstacle in a camera image. If the obstacle 15a, the obstacle 15b, the obstacle (object) 21, and the obstacle 17 are detected within the work range, the detection mark 41a of the obstacle 15a is placed on the video output device (monitor) 1501 and the obstacle 15b. The detection mark 41b, the detection mark 42a of the obstacle (object) 21 and the detection mark 43a of the obstacle 17 are displayed in a circumscribed rectangle of the obstacle. Further, the display is performed such that the millimeter wave radar detection mark is superimposed on the position 41, the position 42, and the position 43 detected by the millimeter wave radar 14a, and the text “There is an obstacle 46” is displayed. Further, the voice output device (speaker) 1502 outputs “There is an obstacle” as a voice.

  FIG. 21 is an explanatory diagram showing an example of a person warning alarm and a warning warning mark 1406 for a person. The obstacle 15a and the obstacle 15b in the work range are persons (workers), and since the obstacle (person) 15a is closest to the vehicle body 1B, there is a possibility of contact with the obstacle (person) 15a. . Therefore, since the alarm level for the obstacle (person) 15a becomes the highest during excavation, the warning alarm mark 1406 is, for example, a thick rectangle for the obstacle (person) 15a, such as “danger: person near 47”. The most noticeable display is given to the operator to alert them. Even with voice, a loud sound “Danger: There is a person nearby” is output. Next, since the obstacle (person) 15b is close to the vehicle body 1B, a rectangle 41b or the like is displayed on the 15b as the caution alarm mark 1406, and the obstacle (object) 21 or the object 17 is away from the vehicle body 1B. The display may not be particularly conspicuous like the rectangle 43a. As a result, the driver can easily confirm that the person is present by approaching the vehicle body 1B, so that the safety of the person can be efficiently grasped, and determination to avoid the backward movement of the vehicle body 1B can be made. it can.

  FIG. 22 is an explanatory diagram of an example of a person contact warning, a contact mark on a contact person, and a machine operation stop signal mark 1408. The obstacle 15d and obstacle 15e in the work range are persons (workers), and the person area 41d is closest to the vehicle body 1B and next to the person area 41e, but in the shooting scene of the camera 13b, the front work machine 1A Therefore, the calculation is performed using the direction so that the contact risk coefficient α of the person 15e is larger than the coefficient α of the person 15d. Therefore, the alarm level for the person 15e is the highest, and the person contact alarm mark 1408 is displayed with the most noticeable display on the operator, such as a thick star mark 41e on the 15e, "Danger: Persons in the turning direction and turning stop 48a, 48b". Call attention. Even with voice, a loud sound “Danger: There is a person in the turning direction and stops turning” is output. Next, since the person 15d contacts the front work machine 1A, for example, a thick rectangular frame 41d is displayed as the person contact warning mark 1408.

  Here, the work ranges 50e and 50f are determined from the machine state capturing means 700 at the time of turning 53. Therefore, since it always changes following the turning state of the front work machine 1A, it is possible to improve the detection accuracy because only obstacles within an appropriate work range are detected.

  FIG. 23 is an explanatory diagram showing a procedure example of the device operation state determination unit 1600. In step 1601, the operation of the machine is determined from the information acquired by the machine state capturing means 700. If the operation is not changed, step 1608 is set to contact avoidance level = 0, and also in the case of excavation, step 1608 is set to contact avoidance level = 0. To do. In the case of reverse, step 1604 determines the reverse camera image, step 1605 calculates the contact risk in the reverse camera image from the contact risk of the contact risk calculation means 1200, and step 1607 reverses. The value of the alarm level is determined using the value of the alarm level determination means 1300 with respect to the direction. If it is determined in step 1607 that the alarm level is 5 or more, step 1610 determines that contact avoidance level = 2, and if the alarm level is less than 5, step 1609 determines that contact avoidance level = 1.

  Next, in the case of turning, a camera image in the turning direction is determined from the turning direction calculated in step 1602, and in step 1603, the contact risk in the turning direction camera image is calculated from the contact risk in the contact risk calculation means 1200. Step 1606 determines the value of the alarm level using the value of the alarm level determination means 1300 with respect to the turning direction. If it is determined in step 1606 that the alarm level is 5 or higher, step 1612 determines that contact avoidance level = 4, and if the alarm level is less than 5, step 1611 determines that contact avoidance level = 3.

  FIG. 24 is an explanatory diagram showing a procedure example of the device control unit 1700. Step 1701 determines the contact avoidance level from the operation state from the device operation state determination means 1600. If the level in step 1701 is 0, step 1702 determines that no device control signal is output (sig = 0).

  If the level of 1701 is 1, step 1703 is determined as a reverse signal for low-speed movement (sig = 1), and if the level of 1701 is 2, step 1704 is determined as a stop signal for reverse movement (sig = 2). To do.

  Further, if the level of 1701 is 3, step 1705 is determined as a turning signal for low-speed movement (sig = 3), and if the level of 1701 is 4, step 1706 is determined as a stop signal for turning (sig = 4). .

  FIG. 25 is an explanatory diagram showing a procedure example of the device control signal output means 1800. In step 1801, a control signal from the contact avoidance control means 1700 is determined. If the control signal is 0 (sig = 0), step 1802 indicates no signal output. When the control signal is 1 (sig = 1) in the determination of step 1801, step 1803 outputs a reverse signal of low speed movement to the lower traveling body 1e and the left and right traveling motors 3e, and the control signal is 2 (sig = 2) Step 1804 outputs a stop signal to the lower traveling body 1e and the left and right traveling motors 3e.

  If the control signal is 3 (sig = 3) in step 1801, step 1805 outputs a low-speed turning signal to the front work machine 1A. If the control signal is 4 (sig = 4), step 1806 is front work. A stop signal is output to the machine 1A.

  Thereby, it is possible to realize real-time surrounding monitoring that does not reduce work efficiency while avoiding contact with a person.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made.

  Obstacles around the machine can be detected without blind spots, and the operator can check the type of obstacles and the risk of contact at a glance, and control the person so that they do not touch the machine during work. Therefore, it can be applied to a surrounding monitoring system such as a mobile robot.

DESCRIPTION OF SYMBOLS 1A Front work machine 1B Car body 1a Boom 1b Arm 1c Bucket 1d Upper turning body 1e Lower traveling body 1f Driver's cab 3a-3c Hydraulic actuator 3e Traveling motors 8a, 8b, 8c Angle detector 8e Direction sensor 13a, 13b Obstacle detector ( camera)
14a Millimeter wave radar 15a, 15b Obstacle (person)
17 Color cone 21 Obstacle (object)
41, 42, 43 Positions 50b, 50d, 50e, 50f Work range 270 Feature calculation unit 600 Obstacle detection unit 700 Machine state capturing unit 1100 Positional calculation unit between obstacles and machine 1200 Contact risk calculation unit 1300 Alarm level determination unit 1400 Alarm output means 1500 Output device 1501 Video output device 1502 Audio output device 1600 Equipment operation state determination means 1700 Contact avoidance control means 1800 Equipment control signal output means

Claims (7)

In the ambient monitoring device that monitors obstacles around the work machine,
An image sensor for detecting image data of the obstacle;
An electromagnetic wave sensor for detecting position data of the obstacle;
Obstacle detection means for detecting the presence of an obstacle such as an operator or an object around a work machine by the image sensor or the electromagnetic wave sensor;
A positional relationship calculation unit for the obstacle detection means for calculating a positional relationship between the working machine and obstacle detection,
If the positional relationship detected by the positional relationship calculating means is within a predetermined distance, a feature calculating means for determining whether the obstacle corresponds to a person based on the presence or absence of a human head feature in the image data;
An operation state capturing means for capturing an operation state of the work machine ;
Contact risk calculation means for calculating a contact risk level according to a determination result as to whether or not the person is determined by the feature calculation means when the operation state capturing means acquires the operation state of the work machine When,
Alarm level determining means for determining an alarm level suitable for the risk of the contact risk calculating means;
A peripheral monitoring device for a work machine, comprising alarm output means for outputting alarm contents to an output device from the determined alarm level. In claim 1,
When the positional relationship between the detected obstacle and the work machine is equal to or greater than a predetermined distance, the feature calculation unit determines whether the obstacle corresponds to a person based on an aspect ratio of the obstacle in the image data. A peripheral monitoring device for a work machine characterized by the above. In claim 1,
When the positional relationship between the detected obstacle and the work machine is greater than or equal to a predetermined distance, the feature calculation means divides the obstacle extraction area in the image data into a head, a trunk, and a lower part, A peripheral monitoring device for a working machine, wherein it is determined whether or not the obstacle corresponds to a person by calculating a contour shape. In claim 1,
The feature calculating means detects the presence or absence of a circular outline as a human head feature in the image data when the positional relationship between the detected obstacle and the work machine is within a predetermined distance. Perimeter monitoring device for machines. In the perimeter monitoring device that monitors obstacles around the work machine,
An image sensor for detecting image data of the obstacle;
An electromagnetic wave sensor for detecting position data of the obstacle;
Obstacle detection means for detecting the presence of an obstacle such as an operator or an object around a work machine by the image sensor or the electromagnetic wave sensor;
A positional relationship calculating means for calculating a positional relationship between the obstacle detected by the obstacle detecting means and the work machine ;
If the positional relationship detected by the positional relationship calculating means is within a predetermined distance, a feature calculating means for determining whether the obstacle corresponds to a person based on the presence or absence of a human head feature in the image data;
An operation state capturing means for capturing an operation state of the work machine ;
Contact risk calculation means for calculating a contact risk level according to a determination result as to whether or not the person is determined by the feature calculation means when the operation state capturing means acquires the operation state of the work machine When,
Alarm level determining means for determining an alarm level suitable for the risk of the contact risk calculating means;
Comprising alarm output means for outputting alarm contents to the monitor from the determined alarm level;
A peripheral monitoring device for a working machine, wherein a different display is performed on a monitor depending on whether or not the obstacle corresponds to a person. In claim 5,
A peripheral monitoring device for a work machine, characterized by displaying a positional relationship between an operation range of the work machine and the obstacle. In claim 5,
A warning monitor for a work machine, wherein the alarm output means displays different information on the obstacle according to the operating state of the work machine.

Images