JP2023145104A - Work machine - Google Patents

Abstract

To accurately identify a person whose body part is partially detected by a range finder as a human.SOLUTION: Provided is a work machine that comprises: a camera that photographs surroundings; a range finder that is disposed so that a range finding visual field overlaps a camera visual field of the camera, and measures a distance to an object within the range finding visual field to acquire coordinate data of a point group on a surface of the object; and a controller that calculates a frame encompassing the object point group range-found by the range finder on a captured image of the camera, and processes a corresponding area of the frame in the captured image to determine whether the object is a human. The work machine determines, based on distribution of the object point group, whether distances to an upper end and a lower end of the object are measured, and if the distance to the upper end or to the lower end is not measured, estimates a non-range-found missing point group of the upper end or lower end of the object based on a predetermined posture and size of a human, and compensates the frame to encompass the range-found object point group and the estimated missing point group.SELECTED DRAWING: Figure 3

Description

The present invention relates to working machines such as hydraulic excavators.

Work machines (hydraulic excavators, etc.) are known that use distance meters (object detection sensors) such as LiDAR or cameras to detect people and objects in the surroundings, perform vehicle control and notifications, and thereby support the operator's driving. There is.

Distance meters cannot distinguish between humans and objects, and both humans and walls and trees are detected as the same three-dimensional object (object). When an object (other than a human being) is detected for which the driving support function should not be activated, the driving support function may be activated more than necessary, which may be troublesome. On the other hand, cameras can distinguish between humans and non-human objects, but the accuracy is generally insufficient, and people whose entire bodies are not visible or whose clothes match the background color may not be detected. It is possible to increase the accuracy of human detection by adjusting the identification parameters of image processing, but in this case, it is easy to falsely detect patterns on the ground, shadows, etc.

For example, there is a known technology that detects people whose whole body cannot fit within the camera's field of view by monitoring the area at the edge of the camera's field of view with a rangefinder and processing the camera image of the area where the object is detected by the rangefinder. (Patent Document 1). Furthermore, a technique is known in which distance information from a distance image sensor is projected onto an image captured by a camera to estimate the position of an object captured by the camera (Patent Document 2).

Patent No. 6689669 JP 2019-4484 Publication

However, with the technology of Patent Document 1, since the overall size of the object cannot be estimated from the object partially detected by the rangefinder at the edge of the camera's field of view, It is necessary to perform image processing on a wide area. If an area wider than necessary is image processed, various things other than the detected object, such as patterns on the ground, will be reflected in the image processing area, making it easy for objects such as walls and trees detected by a rangefinder to be mistaken as humans. .

The distance image sensor with a wide angle of view shown in Patent Document 2 is expensive, so if a distance image sensor with a reduced angle of view is used to reduce the processing load and device cost, it will be possible to image the entire human body with the distance image sensor. There may be no. If the distance image sensor cannot capture the entire body, it is not possible to accurately estimate the extent of the object's presence on the camera image.

An object of the present invention is to provide a work machine that can accurately identify a person whose body part is detected by a rangefinder as a person.

In order to achieve the above object, the present invention includes a vehicle body, a camera that photographs the surroundings of the vehicle body, and a camera that is arranged so that a distance measurement field of view overlaps the camera field of view of the camera, and that measures an object within the distance measurement field of view. a rangefinder that obtains coordinate data of a point group on the surface of the object by distance measurement, and a frame that encloses the object point group, which is the point group of the object surface measured by the rangefinder, on the captured image of the camera. and a controller that processes a corresponding region of the frame in the photographed image to determine whether or not the object is a human, wherein the controller processes the object point group based on the distribution of the object point group. Determine whether the top and bottom ends of the object are measured, and if the top or bottom ends are not measured, determine the missing points on the top or bottom of the object that are not measured based on the preset human posture and size. A working machine is provided that corrects the frame so as to surround the measured object point group and the estimated missing point group.

According to the present invention, a person whose body part is detected by a rangefinder can be identified as a person with high accuracy.

A side view of a hydraulic excavator that is an example of a working machine according to a first embodiment of the present invention A circuit diagram showing the main parts of the hydraulic system installed in the hydraulic excavator shown in Figure 1. Functional block diagram showing the human detection function of the controller installed in the hydraulic excavator in Figure 1 Diagram showing the positional relationship between the rangefinder and the object Diagram showing the positional relationship between the rangefinder and the object A flowchart showing the procedure of defect determination processing by the controller installed in the hydraulic excavator shown in Figure 1. A flowchart showing the procedure of missing point cloud estimation processing by the controller installed in the hydraulic excavator shown in Figure 1. A diagram showing an example of a human model used in the missing point cloud estimation process by the controller installed in the hydraulic excavator in Figure 1. An explanatory diagram of a method for imagining Pxmin that can be taken by a missing point group existing above the rangefinding field of view under the condition that the height of the object point group is greater than the lower half height of the human model. An explanatory diagram of a method for imagining the possible Pxmin of a missing point group existing above the rangefinding field of view under the condition that the height of the object point group is less than the height of the lower half of the human model. An explanatory diagram of a method for imagining the possible Pxmax of a missing point group existing at the lower side of the ranging field of the rangefinder under the condition that the height of the object point group is greater than or equal to the height component of the upper body of the human model. An explanatory diagram of a method for imagining the possible Pxmax of a missing point group existing at the lower side of the rangefinding field of view under the condition that the height of the object point group is less than the height component of the upper body of the human model. Diagram for explaining an overview of object pixel estimation processing by the controller installed in the hydraulic excavator in Figure 1 Diagram for explaining an example of human identification processing by the controller installed in the hydraulic excavator in Figure 1 Diagram showing the positional relationship between the rangefinder and the object A flowchart showing the procedure of defect determination processing by a controller installed in a working machine according to a second embodiment of the present invention. A flowchart showing the procedure of the missing point group estimation process by the controller installed in the working machine according to the second embodiment of the present invention.

Embodiments of the present invention will be described below using the drawings. Although the present invention is applicable not only to hydraulic excavators but also to other types of working machines such as dump trucks, wheel loaders, and cranes, the following description will be given using an example in which the present invention is applied to a hydraulic excavator.

(First embodiment)
-Working machines-
FIG. 1 is a side view of a hydraulic excavator which is an example of a working machine according to a first embodiment of the present invention. In this embodiment, the right and left sides in FIG. 1 are the front and rear of the hydraulic excavator. The hydraulic excavator shown in the figure includes a vehicle body 1 and a front working machine 2 attached to the vehicle body 1. The vehicle body 1 includes a running body 3 and a revolving body 4 provided on the running body 3.

The traveling body 3 is a base structure of a hydraulic excavator, and is a crawler-type traveling body that runs on left and right crawler tracks 5, but a wheel-type traveling body may be used in some cases. The traveling body 3 travels by driving left and right crawler tracks 5 by left and right traveling motors 18 and 19 (FIG. 2), respectively.

The revolving body 4 is provided on the upper part of the traveling body 3 via a revolving wheel 6, and includes a driver's cab 7 in which an operator is seated at the front left side. The driver's cab 7 includes a driver's seat where an operator sits, operating devices such as operating levers for operating various actuators, a monitor 10 (FIG. 3) that displays various data, and the like. A swing motor 17 (FIG. 2) is attached to a swing frame that is a base frame of the swing structure 4. The swing motor is a hydraulic motor, but when an electric motor is used, both a hydraulic motor and an electric motor may be used. A power chamber 8 is provided on the rear side of the driver's cab 7 in the revolving structure 4, and a counterweight 9 is provided at the rearmost portion.

The front working machine 2 is connected to the front part of the revolving body 4 (in this embodiment, to the right side of the operator's cab 7). The front working machine 2 is an articulated working machine that includes a boom 11, an arm 12, and an attachment 13 (a bucket in this embodiment). The boom 11 is directly connected to the revolving frame so as to be vertically rotatable, and is also connected to the revolving body frame via a boom cylinder 14 . The arm 12 is directly rotatably connected to the tip of the boom 11 and is also connected to the boom 11 via an arm cylinder 15 . The attachment 13 is directly rotatably connected to the tip of the arm 12 and is also connected to the arm 12 via an attachment cylinder 16 . The boom cylinder 14, arm cylinder 15, and attachment cylinder 16 are hydraulic cylinders.

Mounted on the upper part of the counterweight 9 are a camera C that photographs the surroundings of the vehicle body 1 (for example, behind the rotating body 4), and a rangefinder (object detection sensor) S2 that acquires a distance image.

The camera C is a camera that has a wide-angle lens and an image sensor (CCD, CMOS, etc.). In some cases, a camera is installed to take pictures of the right side of the vehicle. The camera C is arranged with its optical axis directed diagonally downward so that the side of the vehicle body 1 or the immediate vicinity thereof enters the camera field of view V1 (FIG. 4A).

Although a stereo camera can be used as the range finder R, in this embodiment, it is assumed that it is LiDAR (Light Detection and Ranging). LiDAR is a distance meter that acquires a distance image (coordinate data of a point group) consisting of a plurality of distance measurement points. The rangefinder R is placed close to the camera C so that the parallax with the camera C is small. The rangefinding field of view V2 (FIG. 4A) of the rangefinder R is smaller than (or smaller than) the camera field of view V1 both vertically and horizontally, and the entire rangefinding field of view V2 is included within the camera field of view V1. It overlaps with camera field of view V1. The rangefinder R irradiates a large number of laser beams with optical axis angles shifted in the horizontal direction (yaw direction) and vertical direction (pitch direction) in the distance measurement field of view V2, and calculates the laser beams reflected at each distance measurement point. Measure the distance to each distance measurement point from the time it takes to receive light. With this configuration, the rangefinder R measures the distance to an object (three-dimensional object) existing within the self-range measurement field of view V2 set within the camera field of view V1, and measures a group of points (a plurality of laser irradiation points) on the surface of the object. ) to obtain coordinate data (point cloud data). The coordinate data acquired by the range finder R is, for example, a value in a local coordinate system based on the position of the range finder R.

In the hydraulic excavator shown in FIG. 1, a hydraulic pump 22 (Fig. 2) supplies water to the boom cylinder 14, arm cylinder 15, attachment cylinder 16, swing motor 17 (Fig. 2), and travel motors 18, 19 (Fig. 2). Pressure oil is supplied according to the operation. When the boom cylinder 14, arm cylinder 15, and attachment cylinder 16 are driven by pressure oil, the boom 11, arm 12, and attachment 13 rotate, and the position and attitude of the attachment 13 change. When the turning motor 17 is driven, the rotating body 4 turns. When the traveling motors 18 and 19 are driven, the traveling body 3 travels.

-Hydraulic system-
FIG. 2 is a circuit diagram showing the main parts of the hydraulic system installed in the hydraulic excavator of FIG. 1. The hydraulic system shown in FIG. 2 includes an engine 21, a hydraulic pump 22, directional control valves 24-29, a controller 40, and the like.

The hydraulic pump 22 is a variable displacement pump that discharges pressure oil that drives hydraulic actuators such as the boom cylinder 14. A fixed capacity pump can also be used as the hydraulic pump 22. This hydraulic pump 22 is driven by the engine 21, sucks in hydraulic oil from a tank 23, and discharges pressure oil.

The engine 21 includes a governor 21a that adjusts the amount of fuel injection, and a rotation speed sensor 21b that detects the rotation speed of the engine 21. The engine speed is controlled by the controller 40 controlling the governor 21a based on the engine speed. Moreover, the maximum value of the pressure of the pressure oil discharged from the hydraulic pump 22 is defined by the relief valve 22a.

The directional switching valve 24 is a proportional three-position switching valve that controls the flow (direction and flow rate) of pressure oil supplied from the hydraulic pump 22 to the boom cylinder 14 . Similarly, the directional control valves 25-29 are proportional control valves that control the flow (direction and flow rate) of pressure oil supplied from the hydraulic pump 22 to the arm cylinder 15, attachment cylinder 16, swing motor 17, and travel motors 18, 19, respectively. This is a three-position switching valve. These directional control valves 24-29 are operated by correspondingly provided electromagnetic valves being driven by command signals from the controller 40, and control the oil discharged from the hydraulic pump 22 to drive the corresponding hydraulic actuators. .

-controller-
FIG. 3 is a functional block diagram showing the human detection function of the controller 40. The controller 40 is an in-vehicle computer that includes a processing device such as a CPU that periodically executes calculations and control processes necessary for the human detection function, and a memory (storage device) that stores programs and parameters necessary for the human detection function. It is.

The human detection function calculates a frame F1 (Fig. 10) that encloses a group of points on the object surface measured by the rangefinder R (object point group) on the captured image of the camera, and This function processes the corresponding area and determines whether the measured object is a human. Frame F1 shows the area on the image taken by camera C of the object measured by rangefinder R, and includes the object point group (if there is a missing point group described later) projected on the image taken by camera C. (including missing points). The human detection function of this embodiment has the feature that it can be determined with high accuracy whether or not an object is a human, even if the distance meter R cannot measure the entire object from bottom to top.

To give an overview of the human detection function, first, the controller 40 determines whether the (real) upper and lower ends of the actual object have been measured based on the distribution of the object point group. If it is determined that the upper end or lower end of the object is not measured, the controller 40 moves the object to the object based on the preset human posture and size and on the assumption that the distanced object is a human. The missing points at the upper or lower end of the area that are not measured are estimated and calculated virtually. In this embodiment, it is assumed that the missing point group is a part of an object that is outside the distance measurement field of view V2 of the rangefinder R and is not measured. When the missing point group is virtualized in this way, the controller 40 expands frame F1, which is the range of the object on the captured image of camera C, so that the actually measured object point group and the virtual missing point group are A correction calculation is performed on the frame F1 so that it is surrounded on the image. If it is determined that both the upper end and the lower end of the object have been measured (if it is determined that there is no missing point group), the controller 40 converts the range of the object point group into frame F1 without correction. Calculate as Then, the controller 40 processes the corresponding area of the frame F1 of the image taken by the camera C, determines whether the measured object is a human, and outputs determination data to the monitor 10. Specifically, the human detection function of this embodiment illustrated in this way includes a point cloud generation process 41, an object detection process 42, a defect determination process 43, a defective point group estimation process 44, an object pixel estimation process 45, and a human detection function. It includes an identification process 46.

1. Point cloud generation process In the point cloud generation process 41, the controller 40 generates a rangefinder using the rangefinder R as a reference based on the distance measurement data (vertical angle, horizontal angle, distance) of each point input from the rangefinder R. The coordinates (Xr, Yr, Zr) of each point in the coordinate system (XrYrZr coordinate system) are calculated as point group data. The horizontal angle and horizontal angle of the ranging data are default values for each ranging point, and the distance is a measured value. The vertical angle is a value that increases from the low angle side to the high angle side when viewed from the rangefinder R. The horizontal angle is a value that increases from left to right when viewed from the rangefinder R. The distance is a value that increases as the distance from the rangefinder R increases. The data set of the calculated three-dimensional coordinates of the point group is stored in the memory in association with the distance measurement data on which the calculation is based.

2. Object Detection Process In the object detection process 42, the controller 40 extracts object point group data from the point group data calculated in the point cloud generation process 41. For example, based on the position and orientation of the rangefinder R (known data), point clouds that are estimated to be ground ranging data are excluded from the point cloud data, and the remaining point clouds are clustered according to the distance between the point clouds. One or more point groups obtained through the processing can be extracted as an object point group. Clustering processing can be performed, for example, by finding the closest point for each point in the point group to be processed, and grouping the object point group by assuming that two points within a set distance from each other belong to the same object. .

3. Defect Determination Processing FIGS. 4A and 4B are diagrams showing the positional relationship between the rangefinder R and the object. FIG. 4A shows a state in which the object M straddles the lower edge of the distance measurement field of view V2 of the rangefinder R, and the lower end of the object (or the feet if the object M is a human being) is not measured. FIG. 4B shows a state in which the object M straddles the upper edge of the distance measurement field of view V2 of the rangefinder R, and the upper end of the object (or the head if the object M is a human being) is not measured. In other words, in the state shown in FIG. 4A, there is a part of the object M that is outside the rangefinding field of view V2 below the rangefinder field of view V2 of the rangefinder R, and the point cloud data obtained by the rangefinder R is below the rangefinder field of view V2. lacks some data. On the other hand, in the state shown in FIG. 4B, there is a part of the object M that is outside the rangefinding field of view V2 above the rangefinder field of view V2 of the rangefinder R, and the point cloud data obtained by the rangefinder R is located above the rangefinder field of view V2. Missing some data. In the scene illustrated in FIG. 4A or 4B, even if the camera image of the range of the object point group measured by the rangefinder R is processed, the entire image of the object M is not processed, so what is the object M? is not accurately identified.

Therefore, in the defect determination process 43, the controller 40 determines whether the upper and lower ends of the substance of the object M have been ranged based on the distribution of the object point group measured by the range meter R, that is, whether the object point group is A predetermined algorithm is used to determine whether the distance of the entity M is measured from the bottom end to the top end. In this process, if the number of points constituting the object point group is greater than or equal to the set number on either the upper or lower side of the four sides of the rangefinding field of view V2 of the rangefinder R, the controller 40 straddles the side that includes the set number or more points. It is estimated that a hypothetical part of the object M (missing point group) exists outside the distance measurement field of view V2.

FIG. 5 is a flowchart showing the procedure of defect determination processing 43 by the controller 40.

When the defect determination process 43 is started, the controller 40 reads the point cloud data of the object point group detected (currently ranging) in the object detection process 42 from the memory into the CPU in step S11, and executes the procedure in step S12. Move.

Shifting the procedure to step S12, the controller 40 determines whether the object point group includes a set number or more of points belonging to the lower side of the distance measurement field of view V2 of the range finder R. For example, the controller 40 compares the minimum vertical angle of the distance measurement field of view V2 (the vertical angle of the lower side of the distance measurement field of view V2), which is uniquely determined by the vertical angle of view of the rangefinder R, with the vertical angle of each point of the object point group. do. Then, the controller 40 determines points whose angular difference from the minimum vertical angle is less than or equal to a preset threshold as points belonging to the lower side of the distance measurement field of view V2, and counts the number N1 of these points.

In the subsequent step S13, the controller 40 determines whether the counted number N1 is greater than or equal to a set number Na that is set in advance and stored in the memory, and if N1≧Na, the procedure moves to step S14. , if N1<Na, the procedure moves to step S15.

When the procedure moves to step S14, the controller 40 determines that the missing point group of the object M is below the ranging field of view V2, stores the determination data in the memory, and ends the missing point determination process 43, The process moves to missing point group estimation processing 44 .

When the procedure moves to step S15, the controller 40 determines whether the object point group has a set number or more of points belonging to the upper side of the distance measurement field of view V2 of the range finder R, and moves the procedure to step S16. For example, the controller 40 compares the maximum vertical angle of the ranging field of view V2 (the vertical angle of the upper side of the ranging field of view V2), which is uniquely determined by the vertical angle of view of the rangefinder R, with the vertical angle of each point of the object point group. do. Then, the controller 40 determines points whose angular difference from the minimum vertical angle is less than or equal to a preset threshold as points belonging to the upper side of the distance measurement field of view V2, and counts the number N2 of these points.

In the subsequent step S16, the controller 40 determines whether the counted number N2 is equal to or greater than a set number Nb that is set in advance and stored in the memory, and if N2≧Nb, the procedure moves to step S17. , if N2<Nb, the procedure moves to step S18.

When the procedure moves to step S17, the controller 40 determines that the missing point group of the object M is above the ranging field of view V2, stores the determination data in the memory, ends the missing point determination process 43, and The process moves to point cloud estimation processing 44 .

When the procedure moves to step S18, the controller 40 determines that there is no missing point group of the object M, stores the judgment data in the memory, ends the missing point group estimation process 43, and moves to the missing point group estimation process 44. do.

4. Missing Point Group Estimation Process In the missing point group estimation process 44, if there is a missing point group, the controller 40 detects an object beyond the distance measurement field of view V2 in the photographed image of the camera C based on the preset human posture and size. The maximum range in the horizontal and vertical directions in which M can be reflected is calculated using a predetermined algorithm. This maximum range is the range in which the missing points of the object M can be seen, and the controller 40 corrects the frame F1 based on the calculated maximum range.

FIG. 6 is a flowchart showing the procedure of the missing point group estimation process 44 by the controller 40.

When the missing point group estimation process 44 is started, the controller 40 reads data on the presence/absence and direction of the missing point group from the memory into the CPU in step S21, and moves the procedure to step S22.

In step S22, the controller 40 determines whether it is determined that there is a missing point group in the missing processing determination 43, and if it is estimated that there is a missing point group, the procedure moves to step S23, and no missing point group exists. If it is estimated that this is the case, the procedure moves to step S27.

Moving the procedure to step S23, the controller 40 converts the coordinate data of the object point group from the values (Xr, Yr, Zr) of the rangefinder coordinate system based on the known data of the position and orientation of the camera C and the rangefinder R. The values are converted into camera coordinate system values (Xc, Yc, Zc), and the procedure moves to step S24. The camera coordinate system is a Cartesian coordinate system (FIG. 4A). The Xc axis is parallel to the light receiving surface of the image sensor of camera C and takes a positive value upward when viewed from camera C, and the Yc axis is parallel to the light receiving surface of the image sensor of camera C and takes a positive value pointing rightward as seen from camera C. The Zc axis is perpendicular to the light receiving surface of the image sensor and takes a positive value in the direction away from camera C.

In step S24, the controller 40 estimates the human's upright direction vector (the size is arbitrary) in the camera coordinate system, and moves the procedure to step S25. The direction in which a person stands upright may be, for example, the upward direction of the rotating body 4 (the rotational axis direction of the rotating body 4), or the direction perpendicular to the horizontal plane (vertical direction). In the former case, the upward direction of the rotating body 4 in the camera coordinate system can be calculated from the mounting attitude of the camera C with respect to the rotating body 4. In the latter case, for example, a tilt sensor that detects the inclination angle of the vehicle body 1 with respect to the direction of gravity is provided on the rotating body 4, and the upward direction of the rotating body 4 in the camera coordinate system is calculated based on the inclination of the rotating body 4 with respect to the horizontal plane. can do. An inertial measurement unit (IMU) or the like can be used as the tilt sensor.

In step S25, the controller 40 calculates a virtual plane that passes through the center of gravity of the object point group and is parallel to the plane defined by the human's upright direction vector and the Xc axis of the camera coordinate system, and moves the procedure to step S26. Here, the horizontal axis (rightward) of the virtual plane viewed from camera C is defined as the Xc axis, and the vertical axis (upward) orthogonal to the Xc axis is defined as the Yb axis (FIG. 4A).

In step S<b>26 , the controller 40 virtualizes a missing point group based on the point group data of the object point group, assuming a bent posture or an upright posture of a human in the virtual plane, and performs a missing point group estimation process 44 . , and the process moves to object pixel estimation processing 45. This is because the range in the Xc-axis direction in which a person can be seen by the camera C when bending the waist within the virtual plane is the maximum. The missing point group is virtually calculated based on data of human posture and size (size for each posture) that is set in advance and stored in memory (described later).

When the procedure moves to step S27, the controller 40 sets the missing point group to a blank point group with a point group number of 0, ends the missing point group estimation process 44, and proceeds to the object pixel estimation process 45.

4-1. Virtual method for missing point group As shown in Figure 7, the upper half height of a human being is Lu, the lower half height is Ld, the height is Lh (=Lu+Ld), and the bending point when the human bends the waist is 0, and the maximum bending angle assumed at the bending point O is θmax. Further, in this embodiment, it is assumed that the lower body of the human being is parallel to the Yb axis, that the bending point O is an arbitrary point on the upper body, and that the human being has a straight line shape except for the bending point O. When point A is set at the feet (lower end) of a human being and point B is set at the top of the head (upper end), OA≧Ld and OA+OB=Lh hold. Line segment OA is parallel to the Yb axis.

Here, the memory of the controller 40 stores points Pxmin, Pxmax, Pymax, and Pymin that are set in advance for the model shown in FIG. The point Pxmin is the point with the minimum Xc coordinate that the missing point group can take. The point Pxmax is the point with the maximum Xc coordinate that the missing point group can take. The point Pymax is the point with the maximum Yb coordinate that the missing point group can take. The point Pymin is the point with the minimum Yb coordinate that the missing point group can take. If it is determined in the defect determination process 43 that there is a missing point group above the distance measurement field of view V2 of the rangefinder R, the controller 40 calculates the three points Pxmin, Pxmax, and Pymax to determine which points the missing point group can take. Virtual maximum range. On the other hand, if it is determined in the defect determination process 43 that there is a group of missing points below the distance measurement field of view V2 of the rangefinder R, the controller 40 calculates the three points Pxmin, Pxmax, and Pymin to remove the missing points. Assume the maximum possible range of the group. Details will be explained below.

4-2. When a group of missing points exists above the range-finding field of view A hypothetical method for Pxmin when there is a group of missing points above the range-finding field of view V2 of the rangefinder R will be described.

FIG. 8A is a diagram illustrating a method for imagining Pxmin that can be taken by a missing point group existing above the ranging field of view V2 of the rangefinder R under the condition that the height of the object point group is greater than the height of the lower half of the human model. It is a diagram. The figure shows a human model whose body is bent above the waist (for example, at the back). Pxmin corresponds to point B (the top of the human model's head). The dotted line frame shown in FIG. 8A is the object point group existence range that surrounds all the object point group projected onto the virtual plane without excess or deficiency. This object existence range is defined by a rectangle consisting of two sides parallel to the Xc axis and two sides parallel to the Yb axis. The four sides of the object point group existence range are circumscribed by the point with the minimum Xc coordinate, the point with the maximum Xc coordinate, the point with the minimum Yb coordinate, and the point with the maximum Yb coordinate among the object points projected onto the virtual plane. .

Here, the length of the object point group existence range in the Yb-axis direction (the height of the object point group) is H, the maximum and minimum values of the Xc coordinates of the object point group are Xcmax and Xcmin, and the maximum and minimum values of the Yb coordinate. Let the values be Ybmax and Ybmin. In this case, the Yb coordinate of the human model's feet (point A) is Ybmin, and the upper side of the object existing range intersects the line segment OA or the line segment OPxmin. The upper side of the object existence range is a line segment connecting the points (Xcmin, Ybmax) and (Xcmax, Ybmax). Pxmin can be calculated using a human model whose body is bent by θmax in the negative direction (left side) of the Xc axis, with point A at the lower left corner (Xcmin, Ybmin) of the object existence range. Regarding the bending point O, the line segment OA is parallel to the Yb axis, OA + OPmin = Lh, and the line segment OA or OPmin intersects the upper side of the object existence range, so the upper left corner of the object existence range (Xcmin , Ybmax), Pxmin can be virtualized. From the above, the coordinates of Pxmin can be calculated as (Xcmin-(Lh-H) sin(θmax), Ybmax+(Lh-H) cos(θmax)).

FIG. 8B explains a method of imagining Pxmin that can be taken by the missing point group existing above the ranging field of view V2 of the rangefinder R under the condition that the height of the object point group is less than or equal to the lower half height of the human model. This is a diagram for In this case as well, Pxmin is calculated using a human model in which point A is taken as (Xcmin, Ybmin) and the body is bent by θmax in the negative direction of the Xc axis. Regarding the bending point O, considering that the line segment OA is parallel to the Yb axis and that the line segment OA is greater than or equal to the lower half height Ld, Pxmin can be obtained by taking the bending point O at the waist position of the human model (Xcmin, Ybmin + Ld). can be virtualized. From the above, the coordinates of Pxmin can be calculated as (Xcmin-Lu×sin(θmax), Ybmin+Ld+Lu×cos(θmax)).

The coordinates of Pxmax when there is a group of missing points above the range-finding field of view V2 can also be calculated in the same manner as the coordinates of Pxmin above. When H>Ld, the coordinates of Pxmax are calculated as (Xcmax+(Lh-H) sin(θmax), Ybmax+(Lh-H) cos(θmax)). When H≦Ld, the coordinates of Pxmax are calculated as (Xcmax+Lu×sin(θmax), Ybmin+Ld+Lu×cos(θmax)).

The coordinates of Pymax when there is a missing point group above the distance measurement field of view V2 are obtained assuming a person in an upright posture, so they can be obtained by substituting 0 for θmax in the coordinates of Pxmin or Pxmax. . Specifically, the coordinates of Pymax are calculated as (Xcmin, Ybmin+Lh).

4-3. When there is a missing point group on the lower side of the distance measurement field of view. Figure 9A shows the distance under the condition that the height of the object point group is greater than or equal to the height component of the upper body of the human model (H≦Lu×cos(θmax)). FIG. 6 is a diagram for explaining a method of imagining Pxmax that can be taken by a group of missing points existing below the distance measurement field of view V2 of the meter R; The figure shows a human model with its body bent at the waist. Pxmax corresponds to point A (the foot of the human model). In this figure, it is assumed that the Yb coordinate of point B (the top of the human model's head) is Ybmax, and that the line segment OB or OP intersects the lower side of the object existing range. The lower side of the object existence range is a line segment connecting the point (Xcmin, Ybmin) and the point (Xcmax, Ybmin).

In FIG. 9A, Pxmin can be calculated using a human model whose body is bent by θmax in the negative direction (left side) of the Xc axis. Considering that the inflection point O exists on a straight line passing through point Q (Xcmax, Ybmin) at the lower right corner of the object existing range and point B, and that OP≧Ld, OP=Ld, OB= Pxmax can be assumed under the conditions of Lu. At this time, since the line segment BQ=H/cos(θmax) and OQ=Lu−H/cos(θmax), the coordinates of Pxmax are (Xcmax+Lu×sin(θmax)−Htan(θmax), Ybmin−Lu×cos (θmax)+HLd) can be calculated.

FIG. 9B shows an object point group existing below the rangefinding field of view V2 of the rangefinder R under a constant condition in which the height of the object point group is less than the height component of the upper body of the human model (H>Lu×cos(θmax)). FIG. 6 is a diagram for explaining a method of imagining Pxmax that a group of missing points can take. Similarly to FIG. 9A, in FIG. 9B, a human model is used in which OP=Ld, OB=Lu, and the body is bent by θmax in the negative direction of the Xc axis. The bending point O exists on the right side of the object existing range. The right side of the object existence range is a line segment connecting the points (Xcmax, Ybmax) and (Xcmax, Ybmin). From these conditions, the coordinates of Pxmax in FIG. 9B can be calculated as (Xcmax, Ybmax-Lu×cos(θmax)-Ld).

The coordinates of Pxmmin when there is a group of missing points below the distance measurement field of view V2 can also be calculated in the same manner as the coordinates of Pxmax above. When H≦Lu×cos(θmax), the coordinates of Pxmin are calculated as (Xcmin−Lu×sin(θmax)+Htan(θmax), Ybmin−Lu×cos(θmax)+HLd). When H>Lu×cos(θmax), the coordinates of Pxmin can be calculated as (Xcmin, Ybmax−Lu×cos(θmax)−Ld).

The coordinates of Pymin when there is a missing point group on the lower side of the ranging field of view V2 can be found by assuming a person in an upright posture, so it can be found by substituting 0 for θmax in the coordinates of Pxmin or Pxmax. can. Specifically, the coordinates of Pymin can be calculated as (Xcmin, Ybmax-Lh).

In this embodiment, an example has been described in which only three points among Pxmin, Pxmax, Pymin, and Pymax are calculated in order to calculate a rectangular frame in the object pixel estimation process 45 (described later). However, if the shape of the frame is a polygon with more corners than a rectangle and the virtual accuracy of the range of the human image on the camera image is to be further improved, the number of points to be calculated may be greater than three. Further, Pxmax, etc. may be calculated under the condition that the bending angle of the human model is smaller than θmax, or Pxmax, etc. may be calculated using a human model bent in a direction inclined with respect to the virtual plane.

5. Object Pixel Estimation Process FIG. 10 is a diagram for explaining an overview of the object pixel estimation process 45 by the controller 40. In the object pixel estimation process 45, the controller 40 generates an image of the camera C corresponding to the object point group detected in the object detection process 42 and the missing point group (for example, Pxmin, Pxmax, Pymax) virtualized in the missing point group estimation process 44. Calculate multiple pixels of the sensor. Then, the controller 40 extracts the pixel with the minimum horizontal position, the maximum pixel with the horizontal position, the minimum pixel with the vertical position, and the maximum pixel with the maximum vertical position from the plurality of pixels. , a rectangular frame F1 is calculated. Pixels corresponding to the point cloud can be obtained by using perspective projection transformation of the three-dimensional point cloud using a known pinhole model, for example, based on known data on the positions and orientations of the camera C and the rangefinder R, and internal parameters of the camera C. I can do it. Needless to say, frame F1 includes a vertical line (left side) passing through the pixel with the smallest horizontal position, a vertical line (right side) passing through the pixel with the largest horizontal position, and a vertical line passing through the pixel with the smallest vertical position. It is defined by a horizontal line (lower side) and a horizontal line (upper side) passing through the pixel with the largest vertical axis position.

6. Human Identification Process When proceeding to the human identification process 46, the controller 40 uses the frame F1 calculated in the object pixel estimation process 45 to process the image taken by the camera C to perform human identification. For example, feature extraction such as deep learning and HOG can be used to identify humans. The judgment data of the human identification process 46 is outputted from the controller 40 to the monitor 10, and is displayed as a live view image obtained by combining the frame F1 with the image taken by the camera C, or other appropriate notification forms such as text information or alarm sound. The presence is notified to the operator. As a result of the human identification process 46, for example, if it is determined that there is a human within the maximum turning radius of the hydraulic excavator, in addition to or in place of the notification operation, the solenoid valves of the directional control valves 24-29 are controlled by the controller 40, It is also possible to configure each actuator of the hydraulic excavator to be braked.

Here, as an example of the human identification process 46, the controller 40 processes a cutout image cut out from the image taken by the camera C in the frame F1, and determines whether the object measured by the rangefinder R is a human. can be mentioned. When the cutout image of frame F1 is used for human identification, since the image to be processed is small, the load placed on the controller 40 in the human identification processing 46 is reduced.

Further, as another example of the human identification processing 46, there is also a mode in which the entire image taken by the camera C is processed to identify a human, and the position and size of the identified human are compared with the frame F1 to evaluate the degree of matching. Can be mentioned. An explanatory diagram of this example is shown in FIG. In the example shown in the figure, a person N (real image) is shown in the center of the camera field of view V1. The entire captured image (camera field of view V1) of the camera C is processed by the controller 40, and the image area (dotted chain line) in which it is determined that a human being is shown is defined as a human identification image area F2. For example, this human identification image area F2 is compared with the frame F1, and if the degree of coincidence (overlapping ratio, etc.) of the human identification image area F2 with the frame F1 is greater than or equal to a set value, the measured object M in the frame F1 is a human being. It can be determined that If the degree of coincidence of the human identification image area F2 with the frame F1 is less than the set value, it is determined that the object M measured within the frame F1 is not a human. With only image processing, patterns on the ground, shadows, etc. may be mistaken for humans, but by comparing the result of image processing with frame F1 in this way, humans can be identified with high accuracy.

At this time, generally when identifying a human by image processing, the accuracy of human estimation is calculated, and a conclusion on human identification may be made depending on whether the accuracy is above a certain level. In the case of this embodiment, for example, even for objects for which the accuracy of identifying as a human being is insufficient as a result of image processing alone, if the degree of matching with frame F1 is above a certain level, the object is determined to be a human. Algorithms can also be used.

-effect-
(1) In this embodiment, it is determined whether the object point group detected by the rangefinder R is obtained by measuring the object M from top to bottom. Assuming that is a human, a group of missing points that are outside the distance measurement field of view V2 and are not measured are virtually calculated using a predetermined algorithm. Then, the frame that serves as a reference for the region for identifying humans on the camera image is corrected so as to surround not only the object point group but also the missing point group. According to the present embodiment, by imagining the missing point group and calculating the range (frame) of a person that can be seen in the camera image beyond the rangefinding field of view V2 of the rangefinder R, the rangefinder R can detect a part of the body. A detected human can be identified as a human with high accuracy.

(2) In the present embodiment, if there are more than a set number of points constituting the object point group on either the upper or lower side of the four sides of the rangefinding field of view V2 of the rangefinder R, the controller 40 It is estimated that a group of missing points exists outside the ranging field of view V2 of the range finder R across the included sides. Then, the controller 40 calculates the maximum range in the horizontal and vertical directions in which a human can be seen beyond the rangefinding field of view V2 of the rangefinder R in the captured image of the camera C based on the preset human posture and size. and correct the frame based on the maximum range. With such an algorithm, when the object M is a human being, it is possible to calculate a range (frame) in which the human can be seen without too much or too little. This prevents the human recognition accuracy from decreasing due to the frame being too small and making it impossible to identify whether the object M is a human, or the frame being unnecessarily large and containing unnecessary information, thereby improving the human identification system. can be improved.

(Second embodiment)
A second embodiment of the present invention will be described. The present embodiment is applied to a scene where a group of missing points exists at the bottom of an object that enters the blind spot above the object and is not measured by the rangefinder R, as shown in FIG. The human detection function of this embodiment described below can be implemented in the controller 40 together with the human detection function described in the first embodiment, or can be implemented alone in the controller 40.

In the human detection function in this embodiment, the controller 40 determines that the vertical overlap of the object point group with the ground point group measured by the range meter R is less than a set value, and the minimum distance between the object point group and the ground is If the distance is longer than the set distance, it is estimated that there is a group of missing points hidden in the blind spot. Similarly to the first embodiment, the human detection function of this embodiment includes a point cloud generation process 41, an object detection process 42, a defect determination process 43, a missing point group estimation process 44, an object pixel estimation process 45, and a human identification process 46. Contains. Of these, the point cloud generation process 41, object pixel estimation process 45, human identification process 46, and the hardware configuration of the work machine are the same as in the first embodiment. This embodiment differs from the first embodiment in object detection processing 42, defect determination processing 43, and defective point group estimation processing 44. The object detection process 42, defect determination process 43, and defective point group estimation process 44 of this embodiment will be described below.

-Object detection processing-
In the object detection process 42, the controller 40 extracts, from the point cloud data calculated in the point cloud generation process 41, a ground point group obtained by measuring a distance from the ground in addition to the object point group. The ground point group is, for example, a point group that is estimated to be ground ranging data based on the position and orientation (known data) of the range finder R. Extraction of the object point group is the same as in the first embodiment.

-Deficiency determination processing-
In the scene illustrated in FIG. 12, even if the camera image of the range of the object point group measured by the rangefinder R is processed, the lower end of the object M enters the blind spot and the entire image is not processed. It is not precisely identified what the Therefore, in the defect determination process 43, the controller 40 determines whether the upper and lower ends of the substance of the object M have been ranged based on the distribution of the object point group measured by the range meter R, that is, whether the object point group is A predetermined algorithm is used to determine whether the distance of the entity M is measured from the bottom end to the top end. In this embodiment, the controller 40 determines the presence or absence of the lower end of the object M (missing point group) that is hidden in a blind spot and is not measured from a distance, based on the degree of vertical overlap between the object point group and the ground point group.

FIG. 13 is a flowchart showing the procedure of defect determination processing 43 by the controller 40 in the second embodiment.

When the defect determination process 43 is started, the controller 40 reads point cloud data of the object point group and the ground point group detected (currently ranging) in the object detection process 42 from the memory into the CPU in step S31. The procedure moves to S32.

When the procedure moves to step S32, the controller 40 estimates the human's upright direction vector (the size is arbitrary) in the camera coordinate system, and moves the procedure to step S33. This procedure is similar to step S24 described in the flowchart of FIG. 6 of the first embodiment.

Moving the procedure to step S33, the controller 40 projects the object point group and the ground point group onto a virtual plane (horizontal plane) orthogonal to the upright direction vector, and defines an area surrounding the object point group and the ground point group on the virtual plane. The overlapping area with the enclosing area is calculated, and it is determined whether the overlapping area is greater than or equal to the set area. The set area is a preset value. If the overlapping area is greater than or equal to the set area, the controller 40 moves the procedure to step S37, and if the overlapping area is less than the set area, the controller 40 moves the procedure to step S34.

Note that in step S33, for example, the ratio of the overlapping area to the area of the area surrounding the object point group on the virtual plane may be calculated, and it may be determined whether the ratio is equal to or greater than a set ratio.

When the procedure moves to step S34, the controller 40 extracts a point (described as the lowest point) in which the upright direction component has the minimum value among the object point group, and extracts the horizontal distance from this lowest point (upright direction component). The point of the ground point group with the minimum distance (distance taken in the direction perpendicular to the direction vector) is extracted. The controller 40 calculates the distance between the two extracted points in the upright direction as the minimum distance between the object point group and the ground, and moves the procedure to step S35.

In subsequent step S35, the controller 40 determines whether the minimum distance between the object point group and the ground is equal to or greater than a preset distance, and if it is equal to or greater than the preset distance, the process proceeds to step S36, and if it is less than the preset distance, the controller 40 The procedure moves to step S37.

When the procedure moves to step S36, the controller 40 determines that the lower end of the object M is missing because it is in the upper blind spot, and that the missing point group is at the lower end of the object M, and stores the determination data in the memory. The defect determination process 43 is then terminated, and the process proceeds to the defective point group estimation process 44.

When the procedure moves to step S37, the controller 40 determines that there is no missing point group of the object M, stores the judgment data in the memory, ends the missing point group estimation process 43, and moves to the missing point group estimation process 44. do. This procedure is similar to step S18 described in the flowchart of FIG. 5 of the first embodiment.

- Missing point group estimation processing -
When proceeding to the missing point group estimation process 44, if there is a missing point group, the controller 40 determines whether the missing point group is hidden above the object M in the captured image of the camera C and is not measured based on the preset human posture and size. A point group is assumed and a frame F1 including the missing point group is calculated.

FIG. 14 is a flowchart showing the procedure of the missing point group estimation process 44 in the second embodiment.

When the missing point group estimation process 44 is started, the controller 40 reads the presence/absence and direction of the missing point group determined by the missing point group 43 and the data of the object point group from the memory into the CPU in step S41, and moves the procedure to step S42. .

In step S42, the controller 40 determines whether it is determined that there is a missing point group in the missing processing determination 43, and if it is estimated that there is a missing point group, the procedure moves to step S43, and no missing point group exists. If it is estimated that this is the case, the procedure moves to step S45.

When the procedure moves to step S43, the controller 40 estimates the human's upright direction vector (the size is arbitrary) in the camera coordinate system, and moves the procedure to step S44. This procedure is similar to step S24 described in the flowchart of FIG. 6 of the first embodiment.

In step S44, the controller 40 virtualizes a missing point group based on the point group data of the object point group, assuming the posture of a person standing upright on the ground, ends the missing point group estimation process 44, and processes the object pixel estimation process. 45. The missing point group is virtualized at a position lower than the upper point of the object point group by the human height Lh, which is set in advance and stored in the memory.

When the procedure moves to step S45, the controller 40 sets the missing point group to a blank point group with a point group number of 0, ends the missing point group estimation process 44, and proceeds to object pixel estimation process 45.

-effect-
In this embodiment, even if the feet are in the blind spot of the upper body and the distance cannot be measured by the range meter R, this can be estimated and the frame F1 can be calculated to accurately identify the person as a human. Furthermore, the accuracy of human identification processing can be improved by assuming that the measured object M is a human being who is hidden in the blind spot of the upper body and whose feet are not visible.

Furthermore, by combining this embodiment with the first embodiment, it is possible to include people who are out of the distance measurement field of view V2 and whose parts of their bodies are not measured, and people who are near the hydraulic excavator and whose feet are hidden behind their upper bodies and whose feet are not measured. , both can be identified with high accuracy.

1...Vehicle body, 7...Driver's cabin, 40...Controller, C...Camera, F1...Frame, M...Object, N...Human, N1, N2...Number of points included in the four sides of the ranging field of view, Na, Nb...Number of settings, Pxmax, Pxmin, Pymax, Pymin,...missing point group, R...rangefinder, V1...camera field of view, V2...rangefinding field of view

Claims (4)

The car body and
a camera that photographs the surroundings of the vehicle;
a rangefinder that is arranged so that a distance measurement field of view overlaps the camera field of view of the camera, and that measures an object within the distance measurement field of view to obtain coordinate data of a point group on the surface of the object;
A frame is calculated that encloses the object point group, which is a point group on the object surface measured by the distance meter, on the captured image of the camera, and the corresponding area of the frame in the captured image is processed to determine whether the object is a human being. In a working machine equipped with a controller that determines whether or not there is a
The controller includes:
Based on the distribution of the object point group, determine whether the upper end and the lower end of the object are measured;
If the upper end or the lower end is not ranged, estimate a group of missing points at the upper or lower end of the object that are not ranged based on a preset human posture and size;
A work machine characterized in that the frame is corrected so as to surround the distance-measured object point group and the estimated missing point group. The working machine according to claim 1,
The missing point group is a part of the object that is outside the rangefinding field of view of the rangefinder and is not ranged,
When the number of points constituting the object point group is greater than or equal to a set number on any of the four sides of the rangefinding field of view of the rangefinder, the controller is configured to calculate the distance by straddling sides that include the number of points or more than the set number. A working machine characterized in that it is estimated that the missing point group exists outside the distance measuring field of the meter. The working machine according to claim 2,
The controller calculates the maximum range in the horizontal and vertical directions in which the human can be seen beyond the field of view of the rangefinder in the image taken by the camera, based on a preset posture and size of the human; A working machine, wherein the frame is corrected based on the calculated maximum range. The working machine according to claim 1,
The missing point group is a lower end of the object that falls into a blind spot above the object and is not ranged by the rangefinder,
When the vertical overlap of the object point group with the ground point group measured by the rangefinder is less than a set value, and the minimum distance between the object point group and the ground is equal to or greater than the set distance, the controller A work machine characterized by estimating the existence of a missing point group.

Images