JP6965050B2 - Guide information display device and crane equipped with it and guide information display method - Google Patents

Description

The present invention relates to a guide information display device that assists a crane operation by an operator, and a technique of a crane and a guide information display method provided with the guide information display device.

Conventionally, there has been known a technique for checking the suspended load and the state around the suspended load based on the guide information displayed on the monitor to improve the efficiency of the crane work. Such a technique is disclosed in, for example, Patent Document 1 below.

Patent Document 1 discloses a technique related to a height information notification system that notifies the height of an object around a suspended load. In the height information notification system of an object around the suspended load described in Patent Document 1, the distance from the tip of the boom to the periphery of the suspended load is measured by a distance measuring means such as a laser distance sensor, a microwave distance sensor, and a stereo camera. Then, the position of the object (feature) around the suspended load is detected using the measurement result of the distance, the height is calculated, and the position and height of the object around the suspended load are corresponded to the captured image captured by the camera. The processed image (guide information) is created and the guide information is displayed on the monitor.
In such a guide information display device (height information notification system) described in Patent Document 1, the suspended load can be lifted by transmitting the position information and height information of the suspended load and objects around the suspended load to the operator by the guide information. It suppresses contact with surrounding objects.

Japanese Unexamined Patent Publication No. 2013-120176

However, the guide information display device described in Patent Document 1 measures not only the distance from the tip of the boom to the periphery of the suspended load but also the distance to the wire rope when the wire rope is in the measurement range of the distance measuring means. Will be done. In addition, the guide information display device causes the monitor to display guide information including position information of features around the suspended load based on the measurement result of the distance measuring means. Therefore, in the guide information display device, when the wire rope is present in the measurement range of the distance measuring means, the guide information about the feature is affected by the measurement result of the wire rope, and the position of the suspended load and the object around the suspended load is affected. In some cases, the position information of the wire rope was mixed and displayed in the information. In other words, with the conventional guide information display device, depending on the attitude and operating conditions of the crane, the operator may not be able to accurately grasp the shape of the suspended load and the features existing around it based on the guide information. there were.

The present invention has been made in view of the current problems, and accurate information on the shape and position of the suspended load and the features existing around the suspended load is obtained regardless of the posture and operating conditions of the crane. It is an object of the present invention to provide a guide information display device which can be presented to.

The problem to be solved by the present invention is as described above, and next, the means for solving this problem will be described.

That is, the guide information display device according to the present invention is a guide information display device for a crane including a camera, a laser scanner, a data processing unit, and a data display unit, and the camera is a work area of the crane. The laser scanner acquires the point group data of the work area photographed by the camera, and the data processing unit obtains the point group data acquired by the laser scanner by the crane. Exclude the point group data existing between the suspended load and the tip of the boom of the crane, and based on the remaining point group data, the upper surface of the suspended load and the ground surface of the work area. , The upper surface of the feature existing in the work area is estimated, and a guide frame image surrounding the upper surface of the suspended load and the upper surface of the feature is generated, and the image captured by the camera is generated by the data processing unit. The guide frame images are superposed and displayed on the data display unit.

Further, in the guide information display device according to the present invention, the data processing unit has a predetermined elevation difference from an arbitrary elevation value between the suspended load suspended by the crane and the tip of the boom of the crane. The point group data inside is estimated to be on the same plane to form a plane cluster, and a straight line connecting the center of gravity of the suspended load and the suspended position of the suspended load at the tip of the boom is calculated. The width of the suspended load from the position of the center of gravity of the suspended load to the outer periphery of the suspended load, which is the most distant from the position of the center of gravity of the suspended load in a plan view from a linear direction, is calculated, and the horizontal distance between the center of gravity of the plane cluster and the straight line is calculated. When the horizontal distance is smaller than the suspended load width, or when the suspended load and the planar cluster overlap in a plan view from the vertical direction, the point group data forming the planar cluster is acquired by the laser scanner. It is the point group data to be excluded from the point group data.

Further, in the guide information display device according to the present invention, the data processing unit sets an exclusion region between the suspended load and the tip of the boom, and the point cloud data acquired in the exclusion region is used as the laser. It is the point cloud data to be excluded from the point cloud data acquired by the scanner.

Further, the crane according to the present invention is provided with a guide information display device.

Further, the guide information display method of the guide information display device according to the present invention includes a camera that captures a part of the work area of the crane, a laser scanner that acquires point group data from above the suspended load in the work area, and a laser scanner. The data acquisition unit that acquires the point group data of the work area taken by the camera by the laser scanner, the upper surface of the suspended load based on the point group data, the ground surface of the work area, and the above. A data processing unit that estimates the upper surface of a feature existing in the work area and generates a guide frame image that surrounds the upper surface of the suspended load and the upper surface of the feature, respectively, and the data processing unit on the image captured by the camera. A guide information display method in a guide information display device including a data display unit that superimposes and displays the guide frame images generated in the above, and the point group data within a predetermined altitude difference from an arbitrary altitude value. A plane cluster acquisition step of acquiring a plane cluster formed by presuming that the two are on the same plane, an outline acquisition step of acquiring an outline having the highest elevation value in the outline of the plane cluster, and the outline. A suspension position calculation step of calculating the suspension position of the suspended load at the tip of the boom of the crane, and a straight line calculation step of calculating a straight line connecting the center of gravity of the suspended load to the suspension position. And the horizontal distance calculation step of calculating the horizontal distance between the center of gravity of the plane cluster and the straight line, and the suspended load whose horizontal distance is the most distant from the position of the center of gravity of the suspended load in the plan view from the straight direction. If it is smaller than the suspended load width, which is the distance to the outer circumference of the It has a plane cluster exclusion step of excluding from the point group data acquired by the scanner.

As the effect of the present invention, the following effects are exhibited.

According to the present invention, it is possible to accurately present information on the shape and position of the suspended load and the features existing around the suspended load regardless of the posture and operating conditions of the crane.

The schematic diagram which shows the whole structure of the crane to which the guide information display device which concerns on one Embodiment of this invention is applied. The plan view which explains the suspended load area. The schematic diagram which shows the whole structure of the guide information display device which concerns on one Embodiment of this invention. The side schematic diagram explaining the suspended load area. Explanatory drawing of camera space coordinate system and angle of view of camera, (A) schematic view of Z-axis direction, (B) schematic view of X-axis direction. Explanatory drawing of laser irradiation state by a laser scanner, (A) schematic view in the X-axis direction, (B) schematic view in the Y-axis direction. A schematic view showing a data acquisition unit, (A) a view looking up in the Y-axis direction, and (B) a cross-sectional view taken along the line AA in FIG. 7 (A). The figure which shows the display state of the guide information, (A) the figure which shows the data display part which displayed the image M, (B) the figure which shows the data display part which superposed and displayed the image M and the guide information GD. A schematic diagram showing another configuration of the guide information display device, (A) when the data processing unit, data display unit, and data input unit are configured by a tablet PC, (B) the data display unit and data input unit are connected by a touch panel display device. When configured. The schematic diagram which shows the relationship between the suspended load area and a laser lateral line. The Z-axis direction schematic diagram explaining the laser irradiation situation by a laser scanner. A flow chart showing the flow of data processing by the data processing unit. A diagram showing the point cloud data acquired by the data acquisition unit, (A) a diagram in which the point cloud data is plotted in the XYZ coordinate system, and (B) a diagram in which the point cloud data plotted in the XYZ coordinate system is divided into a plurality of groups. The schematic diagram which shows the designation state of the ground surface by an operator. An explanatory diagram of a reference height calculation method when the ground surface is specified, (A) a diagram showing a reference height calculation method, and (B) a schematic diagram showing a division status of a suspended load area. Explanatory drawing of the generation method of a plane cluster. A flow chart showing a plane estimation procedure. Explanatory drawing of the plane estimation procedure (STEP-201-202). Explanatory drawing of the plane estimation procedure (STEP-203-205). Explanatory drawing of the plane estimation procedure (STEP-206-208). Explanatory drawing of plane estimation procedure (STEP-206-208 (second time)). Explanatory drawing of plane estimation procedure (top surface estimation situation). Flow diagram of how to join planes that exist in different groups. Explanatory drawing of the method of connecting planes existing in different groups. Explanatory drawing of clustering processing of the same area. Explanatory diagram of hierarchical clustering, (A) the case of the feature of the first example, (B) the case of the feature of the second example. Explanatory drawing of guide frame generation procedure. The figure which shows the setting example of the same area cluster in hierarchical clustering, (A) when it is the same area cluster that surrounds the whole, (B) when it sets the plane cluster with the highest elevation value as another same area cluster, (C ) When setting another same area cluster including all planar clusters whose elevation value difference is equal to or greater than the threshold value. The schematic diagram which shows the alarm display in a data display part. A flow chart showing the flow of deleting a plane cluster by the data processing unit. Explanatory drawing of deletion of planar cluster by data processing unit, (A) explanatory view of deletion of planar cluster between suspended load and tip of boom, (B) explanatory view of Y-axis direction view. Explanatory drawing of setting situation of exclusion area.

Next, embodiments of the invention will be described.
As shown in FIG. 1, the crane 1 is an example of a crane to which the guide information display device according to the embodiment of the present invention is applied, and is a mobile crane that can move to a desired place.
The crane 1 includes a traveling vehicle 10 and a crane device 20.

The traveling vehicle 10 conveys the crane device 20, has a plurality of (four in this embodiment) wheels 11, and travels by using an engine (not shown) as a power source.
Outriggers 12 are provided on the four sides of the traveling vehicle 10. The outrigger 12 is composed of an overhang beam 12a that can be extended by hydraulic pressure on both sides of the traveling vehicle 10 in the width direction and a hydraulic jack cylinder 12b that can be extended in a direction perpendicular to the ground. Then, the traveling vehicle 10 can bring the crane 1 into a workable state by grounding the jack cylinder 12b, and by increasing the extension length of the overhang beam 12a, the workable range (work) of the crane 1 is increased. Radius) can be expanded.

The crane device 20 lifts the suspended load W with a wire rope, and has a swivel 21, a telescopic boom 22, a main hook block 23, a sub hook block 24, an undulating cylinder 25, a main winch 26, a main wire rope 27, and a sub winch. It includes 28, a sub-wire rope 29, and a cabin 30.

The swivel base 21 is configured so that the crane device 20 can be swiveled, and is provided on the frame of the traveling vehicle 10 via an annular bearing. The annular bearing is arranged so that its center of rotation is perpendicular to the installation surface of the traveling vehicle 10. The swivel base 21 is configured to be rotatable in one direction and the other direction with the center of the annular bearing as the center of rotation. Further, the swivel table 21 is rotated by a hydraulic swivel motor (not shown).

The telescopic boom 22, which is a boom, supports the wire rope so that the suspended load W can be lifted. The telescopic boom 22 is composed of a base boom member 22a, a second boom member 22b, a third boom member 22c, a force boom member 22d, a fifth boom member 22e, and a top boom member 22f, which are a plurality of boom members. Each boom member is nested in the order of the size of the cross-sectional area. The telescopic boom 22 is configured to be telescopic in the axial direction by moving each boom member with a telescopic cylinder (not shown). The telescopic boom 22 is provided so that the base end of the base boom member 22a can swing on the swivel base 21. As a result, the telescopic boom 22 is configured to be horizontally rotatable and swingable on the frame of the traveling vehicle 10.

The main hook block 23 is for hooking and suspending the suspended load W, and is provided with a plurality of hook sheaves around which the main wire rope 27 is wound and a main hook 32 for suspending the suspended load W.
In addition to the main hook block 23, the crane device 20 further includes a sub-hook block 24 for hooking and suspending the suspended load W, and the sub-hook block 24 is provided with a sub-hook 33 for suspending the suspended load W. Has been done.

The undulating cylinder 25 raises and lays down the telescopic boom 22 to maintain the posture of the telescopic boom 22. The undulating cylinder 25 is composed of a hydraulic cylinder including a cylinder portion and a rod portion.

The main winch 26 is for feeding (winding up) and feeding out (rolling down) the main wire rope 27, and is configured by a hydraulic winch in the present embodiment.
The main winch 26 is configured such that the main drum around which the main wire rope 27 is wound is rotated by a main hydraulic motor. The main winch 26 pays out the main wire rope 27 wound around the main drum by supplying hydraulic oil so that the main hydraulic motor rotates in one direction, and the main hydraulic motor rotates in the other direction. By supplying the hydraulic oil in this way, the main wire rope 27 is wound around the main drum and fed in.

Further, the sub winch 28 is for feeding and feeding the sub wire rope 29, and in the present embodiment, it is configured by a hydraulic winch.

The cabin 30 covers the driver's seat 31 on which the operator sits, and is provided on the side of the telescopic boom 22 on the swivel table 21.

The crane 1 configured in this way can move the crane device 20 to an arbitrary position by traveling the traveling vehicle 10, and the telescopic boom 22 is erected at an arbitrary undulating angle by the undulating cylinder 25. As a result, the telescopic boom 22 can be extended to an arbitrary telescopic boom length.

Further, the crane 1 includes a controller 34 that controls the operation of the swivel base 21, the telescopic boom 22, the undulating cylinder 25, and the like (that is, the operation of the crane 1). The controller 34 can output information related to the operating state of the swivel base 21, the telescopic boom 22, the undulating cylinder 25, etc., information related to the performance peculiar to the crane 1, the weight of the suspended load W, and the like to the outside. ..

In this description, the XYZ coordinate system as shown in FIG. 1 is defined with reference to the axial direction of the undulating fulcrum of the telescopic boom 22 (the same applies to the following description).
The X-axis direction (also referred to as the lateral line direction) is a horizontal direction parallel to the axial direction of the undulating fulcrum of the telescopic boom 22. Further, the Y-axis direction (also referred to as an altitude direction) is a vertical direction. Further, the Z-axis direction (also referred to as the depth direction) is a horizontal direction perpendicular to the axial direction of the undulating fulcrum of the telescopic boom 22. That is, as shown in FIG. 2, the XYZ coordinate system is defined as a local coordinate system based on the telescopic boom 22.

Next, the guide information display device according to the embodiment of the present invention will be described.
The crane 1 includes a guide information display device 50 as shown in FIG.
The guide information display device 50 is an example of the guide information display device according to the present invention, and in order to enable the work by the crane 1 as shown in FIG. 1 to be performed efficiently and safely, the suspended load W This is a device for displaying information (hereinafter, referred to as guide information) of an area including (hereinafter, referred to as a suspended load area WA) as an image and presenting it to an operator.

As shown in FIGS. 2 and 4, the “suspended load area WA” referred to here is set as an area including the suspended load W in the Y-axis direction view within the working area SA of the crane 1. This is the area for which "guide information" is generated.
The "suspended load area WA" is set as an area including directly under the top boom member 22f of the telescopic boom 22 in the crane 1, and the suspended load W, the ground surface F, and the feature C existing in the suspended load area WA guide the crane 1. It is an object to be measured by the information display device 50. The "suspended load area WA" is displaced according to the turning motion, the undulating motion, and the telescopic motion of the telescopic boom 22.

Further, the "guide information" referred to here refers to the operator regarding the quality of the length, turning position, undulation angle, unwinding amount of the wire rope, etc. of the telescopic boom 22 when the operator conveys the suspended load W by the crane 1. Information related to the image information of the suspended load area WA, the information related to the shape of the suspended load W and the feature C, the height information of the suspended load W, the height information of the feature C, and the information of the suspended load W. Information related to the flow line is included.

As shown in FIGS. 3 and 4, the guide information display device 50 is composed of a data acquisition unit 60, a data processing unit 70, a data display unit 80, and a data input unit 90.

The data acquisition unit 60 is a portion that acquires data necessary for generating guide information in the suspended load area WA, and as shown in FIG. 3, includes a camera 61, a laser scanner 62, and an inertial measurement unit (IMU) 63. I have.

As shown in FIG. 4, the data acquisition unit 60 is attached to the top boom member 22f located at the tip of the telescopic boom 22 of the crane 1, and is directly below the boom tip located directly above the suspended load W. It is arranged so that the situation can be grasped. The "directly above" the suspended load W here includes a position vertically above the suspended load W and a position within a certain range (for example, the range of the upper surface of the suspended load W) with respect to the position. It is a concept.

The data acquisition unit 60 is attached to the top boom member 22f at the tip of the telescopic boom 22 via a gimbal 67 (see FIG. 1), and when the telescopic boom 22 performs an undulating operation, a turning operation, or an expansion / contraction operation. In addition, the posture of the data acquisition unit 60 (posture toward the Y-axis direction) can be held substantially constant. As a result, the camera 61 and the laser scanner 62 can always be directed to the suspended load W. Therefore, the data acquisition unit 60 can always acquire data from the suspended load W and the ground surface F (that is, the suspended load region WA) existing below the suspended load W by the camera 61 and the laser scanner 62. Further, when the feature C exists in the suspended load area WA, the data of the feature C can be acquired by the camera 61 and the laser scanner 62.

As shown in FIGS. 5A and 5B, the camera 61 is a digital video camera for shooting an image of the suspended load area WA, and has a function of outputting the photographed image to the outside in real time. .. The camera 61 has an angle of view (horizontal angle of view θh and vertical angle of view θv) as shown in FIGS. 5A and 5B. Further, the camera 61 has a number of pixels, a frame rate, and an image transmission rate in consideration of the amount of data required for generating appropriate guide information.

As shown in FIG. 3, the laser scanner 62 irradiates the object to be measured with a laser and receives the reflected light of the object to be measured by the laser to acquire information related to the reflection point of the object to be measured. It is a device that acquires point group data. The objects to be measured by the laser scanner 62 are a suspended load W, a feature C, and a ground surface F. Further, a first GNSS receiver 65 for acquiring a measurement time is connected to the laser scanner 62.
The guide information display device 50 acquires planar three-dimensional point cloud data in real time by the laser scanner 62.

As shown in FIG. 6, the laser scanner 62 includes a total of 16 laser transmission / reception sensors, and can simultaneously irradiate the measurement target with 16 lasers to acquire point cloud data of the measurement target. It can be done. Each of the 16 laser transmission / reception sensors of the laser scanner 62 is arranged at different irradiation angles by 2 ° in the Z-axis direction, and irradiates the object to be measured with a laser having a spread of 30 ° as a whole. It is configured to be possible. Further, each laser transmission / reception sensor of the laser scanner 62 is configured to be rotatable 360 ° (omnidirectional) around the Z axis. In the following description, the locus drawn by the laser irradiated toward the suspended load region WA is referred to as a laser lateral line. The laser lateral lines are parallel to the X-axis direction, and the laser scanner 62 draws 16 laser lateral lines at the same time.

The laser scanner 62 is arranged so that the laser side line is parallel to the X-axis direction. Further, in the laser scanner 62, the reference axis for changing the laser irradiation angle is parallel to the Z-axis direction.

As shown in FIG. 3, the inertial measurement unit (hereinafter referred to as IMU) 63 is a device for acquiring the attitude data of the camera 61 and the laser scanner 62 at the time of data acquisition. The IMU 63 can measure the attitude angle in real time, and has a measurement accuracy that can be used for correcting the point cloud data acquired by the laser scanner 62. Further, a second GNSS receiver 66 for acquiring the measurement time is connected to the IMU 63.

As shown in FIGS. 7A and 7B, the data acquisition unit 60 is a sensor unit in which the camera 61, the laser scanner 62, and the inertial measurement unit (IMU) 63 are fixed to the frame body 64 and integrally configured. ..
The frame body 64 is a substantially rectangular parallelepiped object formed by combining five plate members. The frame body 64 is composed of four plate materials to form four side surfaces of the rectangular parallelepiped, and the remaining one plate material to form an upper surface portion of the rectangular parallelepiped and has an opening at the bottom. In the data acquisition unit 60, the camera 61 and the laser scanner 62 are attached to the inside of the side surface portion of the frame body 64, and the IMU 63 is attached to the upper surface portion of the frame body 64. As shown in FIG. 7A, the center position of the image pickup element of the camera 61 and the center position of the laser of the laser scanner 62 are separated from each other in the Z-axis direction by a distance Δzh in the Y-axis direction. The laser center position is the rotation center of the laser in the laser scanner 62 and is located on the Z axis.
Further, as shown in FIG. 7B, the center position of the image pickup element of the camera 61 and the center position of the laser of the laser scanner 62 are separated from each other in the Y-axis direction by a distance Δyv in the X-axis direction.
In the data acquisition unit 60, of the four side surface portions of the frame body 64, one of the pair of side surface portions facing each other is perpendicular to the Z axis, and the other of the pair of side surface portions facing each other is perpendicular to the X axis. It is arranged in the posture of. Further, the data acquisition unit 60 is arranged so that the upper surface portion of the frame body 64 is perpendicular to the Y axis.

Next, a method of converting the X coordinate of the point (x, y) in the XYZ coordinate system into the Xc coordinate in the camera space coordinate system will be described.

In the guide information display device 50, the coordinate values are converted between the XYZ coordinate system and the camera space coordinate system in order to superimpose the guide information GD described later on the image M captured by the camera 61 and display it on the data display unit 80. Perform processing. In the guide information display device 50, the three-dimensional camera space coordinate systems Xc, Yc, and Zc are defined in the image space of the camera 61.

As shown in FIG. 5A, the distance in the X-axis direction from the perpendicular line extending from the center of the lens of the camera 61 to the point (x, y) is dh, and the maximum screen width in the horizontal direction of the camera 61 is wh. Further, the points (x, y) are defined as x at the position in the X-axis direction from the center of the screen. At this time, the Xc coordinates of the points (x, y) in the camera space are expressed by the following mathematical formulas (1) and (2).
In the following formula, the difference between the image pickup element of the camera 61 and the position of the center of the laser in the horizontal direction is Δzh (see FIG. 7A), the width of the camera image is wh, and the horizontal angle of view of the camera 61 is θh. The temporary variable is tp1.
tp1 = (y−Δzh) × tan (π × θh / 360) ・ ・ ・ (1)
Xc = wh / 2-wh × x / (2 × tp1) ・ ・ ・ (2)

Next, a method of converting the Z coordinate of the point (y, z) in the XYZ coordinate system into the Zc coordinate in the camera space coordinate system will be described.
As shown in FIG. 5B, the distance from the point (y, z) to the center of the laser in the Z-axis direction is dv, and the maximum horizontal screen width of the camera 61 is wv. Further, the point (y, z) is defined as a position in the Z-axis direction from the center of the screen. At this time, the Zc coordinates of the points (y, z) in the camera space are expressed by the following mathematical formulas (3) and (4).
In the following formula, the difference in the vertical direction between the image sensor of the camera 61 and the position of the laser center of the laser scanner 62 is Δyv (see FIG. 7B), the vertical width of the camera image is wv, and the vertical image of the camera 61. The angle is θv and the temporary variable is tp2.
tp2 = Y × tan (π × θv / 360) ・ ・ ・ (3)
Zc = wv / 2 + wv × (Z−Δyv) / (2 × tp2) ・ ・ ・ (4)

The guide information display device 50 uses the above equations (1) to (4) to convert the coordinates of the point cloud data acquired by the laser scanner 62 or the like in the XYZ coordinate system into the camera space coordinate system, thereby converting the camera 61. The guide information GD is aligned and displayed on the image M taken in.

As the laser scanner 62, an apparatus capable of measuring the three-dimensional shape of the object to be measured is selected from the maximum reachable height (for example, about 100 m) in consideration of the maximum reachable height of the telescopic boom 22. Further, as the laser scanner 62, a device having predetermined performance for each specification such as measurement speed, number of measurement points, measurement accuracy, etc. is used in consideration of the amount of data and data accuracy required to generate appropriate guide information. select.

In the present embodiment, a case where a laser scanner 62 having a total of 16 laser transmission / reception sensors is used is illustrated, but the guide information display device according to the present invention includes the number of laser transmission / reception sensors constituting the laser scanner. Not limited to. That is, in the guide information display device according to the present invention, a laser scanner having optimum specifications is appropriately selected according to the maximum reaching height of the boom (jib) of the crane.

The data acquired in the suspended load area WA by the data acquisition unit 60 includes images of the suspended load W, the ground surface F below the suspended load W, and the feature C existing around the suspended load W taken by the camera 61. Contains data. Further, the data acquired by the data acquisition unit 60 in the suspended load area WA includes the point cloud data acquired by scanning the suspended load W, the ground surface F, and the feature C with the laser scanner 62. The ground surface F referred to here broadly includes a surface serving as a transport source and a transport destination of the suspended load W, and includes not only the ground surface but also the floor surface and roof surface of the roof of the building.

As shown in FIG. 3, the data processing unit 70 is a part for processing the data acquired by the data acquisition unit 60 to generate the guide information GD to be presented to the operator, and in the present embodiment, the predetermined data. It consists of a general-purpose personal computer on which the processing program is installed.
Further, the data processing unit 70 is electrically connected to the controller 34 of the crane 1, and the "crane information" output from the controller 34 is input to the data processing unit 70.

The data display unit 80 is a part for displaying the guide information GD presented to the operator, and is composed of a display device connected to the data processing unit 70.
As shown in FIG. 8A, the data display unit 80 displays the image M of the suspended load area WA captured by the camera 61 in real time.

As shown in FIG. 8B, the guide information GD includes a guide frame GD1 which is a guide frame image showing the outer shape of the suspended load W and the feature C in the Y-axis direction, and the height of the lower surface of the suspended load W. Includes information GD2, height information GD3 on the upper surface of the feature C, working radius information GD4 indicating the flow line of the suspended load W, axis information GD5 indicating the axial direction of the telescopic boom 22, and the like.
Then, the guide information GD generated by the data processing unit 70 and the video M are superimposed and displayed on the data display unit 80.

As shown in FIG. 3, the data input unit 90 is a part for inputting a set value or the like to the data processing unit 70, and is composed of a touch panel, a mouse, a keyboard device, and the like.

As shown in FIG. 9A, the guide information display device 50 uses a tablet-type general-purpose personal computer (hereinafter, also referred to as a tablet PC) to perform the data processing unit 70, the data display unit 80, and the data input unit 90. It is preferable to configure them integrally. Further, as shown in FIG. 9B, the guide information display device 50 integrally comprises a data display unit 80 and a data input unit 90 by a touch panel display device, and the touch panel display device is used as a general-purpose PC for data processing. The configuration may be such that the unit 70 is connected.

As shown in FIG. 4, the data display unit 80 and the data input unit 90 are arranged at positions in the cabin 30 in front of the driver's seat 31 so that the operator can easily see them. The data processing unit 70 is preferably arranged in the vicinity of the data acquisition unit 60. When the data processing unit 70, the data display unit 80, and the data input unit 90 are integrally configured by the tablet PC, the data processing unit 70 may be arranged in the cabin 30.
Data transmission between the data acquisition unit 60 and the data processing unit 70 is preferably via a wired LAN. For data transmission between the data acquisition unit 60 and the data processing unit 70, a wireless LAN may be adopted, or a power line communication may be adopted.

As shown in FIG. 9A, the guide information display device 50 uses a tablet-type general-purpose personal computer (hereinafter, also referred to as a tablet PC) to perform the data processing unit 70, the data display unit 80, and the data input unit 90. It is preferable to configure them integrally. Further, as shown in FIG. 9B, the guide information display device 50 integrally comprises a data display unit 80 and a data input unit 90 by a touch panel display device, and the touch panel display device is used as a general-purpose PC for data processing. The configuration may be such that the unit 70 is connected.

Here, the data acquisition status by the data acquisition unit 60 will be described.
The data acquisition unit 60 continuously photographs the suspended load area WA by the camera 61, and acquires the image M of the suspended load area WA.

As shown in FIG. 10, the data acquisition unit 60 continuously scans the suspended load area WA by the laser scanner 62, and acquires the point cloud data of the object to be measured in the suspended load area WA. In the following, the point cloud data acquired by the laser scanner 62 will be referred to as point cloud data P. The point cloud data P is a set of point data p, and the point data p represents a point located on the upper surface of the ground surface F, the suspended load W, and the feature C existing in the suspended load region WA. Then, as shown in FIG. 11, the point data p includes the distance a from the measurement object (for example, the feature C) to the laser scanner 62 and the irradiation angle b of the laser scanner 62 when the point data p is acquired. Information is included.

As shown in FIG. 3, a first GNSS receiver 65 is connected to the laser scanner 62, and at the same time as acquiring point cloud data P, the first GNSS receiver 65 receives time information from a plurality of positioning satellites. do. Then, the data processing unit 70 adds information related to the acquisition time of the point data p to the point data p. That is, the information related to the point data p includes the acquisition time tp in addition to the distance a and the irradiation angle b.

Further, in the data acquisition unit 60, the point cloud data P is acquired by the laser scanner 62, and at the same time, the attitude data Q of the laser scanner 62 is acquired by the IMU 63 at a predetermined cycle. The attitude data Q includes information related to the angle and acceleration of the laser scanner 62 with respect to each of the X, Y, and Z axes. The acquisition cycle of the attitude data Q by the IMU 63 is shorter than the acquisition cycle of the point cloud data P by the laser scanner 62. The posture data Q is a set of individual posture data q measured for each measurement cycle.

A second GNSS receiver 66 is connected to the IMU 63, and at the same time as the attitude data Q is acquired, the second GNSS receiver 66 receives time information from a plurality of positioning satellites. The data processing unit 70 assigns the acquisition time tq to the individual posture data q as information related to the acquisition time of the individual posture data q. That is, the information related to the individual posture data q includes the acquisition time tq.

Next, the data processing status by the data processing unit 70 will be described.

As shown in FIG. 12, in the data processing by the data processing unit 70, first, "frame extraction processing" is performed (STEP-101).
In the data processing of the point cloud data P by the data processing unit 70, the point cloud data P for one frame is cut out from the stream data of the point cloud data P and output. The point cloud data P for one frame is a set of point data p acquired while the laser irradiation direction of the laser scanner 62 makes one round around the Z axis.

As shown in FIG. 12, in the data processing by the data processing unit 70, "synchronization processing of point cloud data and posture data" is then performed (STEP-102).
The data processing unit 70 synchronizes the point data p included in the point cloud data P for one frame with the attitude data Q acquired by the IMU 63.
Specifically, in each point data p, the acquisition time tq of the individual posture data q closest to the acquisition time tp of the point data p is searched, and the individual posture data q at the acquisition time tq is used as the point data p. Synchronize by associating with.

In this way, the data processing unit 70 outputs the point data p synchronized with the individual posture data q.
Then, as shown in FIG. 11, the data processing unit 70 calculates the distance h from the laser center position of the laser scanner 62 to the point data p based on the distance a and the irradiation angle b. The "distance h" referred to here is the distance from the laser center position of the laser scanner 62 to the horizontal plane in which the point data p exists.

Further, when calculating the distance h of the point data p, the data processing unit 70 makes a correction using the individual posture data q corresponding to the point data p. As a result, the error caused by the posture of the laser scanner 62 can be eliminated, and the distance h of the point data p can be calculated more accurately.

That is, in the guide information display device 50, the data acquisition unit 60 includes an IMU 63 that acquires the attitude data Q of the laser scanner 62, and the data processing unit 70 is based on the attitude data Q of the laser scanner 62 acquired by the IMU 63. The point group data P is corrected.
With such a configuration, the guide information display device 50 can present a more accurate guide information GD to the operator.

When the point cloud data P for one frame is plotted on the XYZ coordinate system, it is represented as shown in FIG. 13 (A). FIG. 13A is a point cloud data P (a set of point data p) viewed from the Z-axis direction.

As shown in FIG. 12, in the data processing by the data processing unit 70, the "ground surface estimation process" is then performed (STEP-103). The data processing unit 70 performs a process of estimating the ground surface F.

First, a case where the ground surface F is estimated with reference to a specific position on the image will be described. Although the case where the operator manually specifies a specific position on the video is illustrated here, the data processing unit 70 may automatically determine and specify the specific position on the video.
In the guide information display device 50, the reference ground surface F can be determined by designating the position of the ground surface in the data display unit 80 and the data input unit 90.
In the case of manual operation, first, as shown in the upper figure of FIG. 14, the operator specifies a position clearly on the ground surface on the image displayed on the data display unit 80. Then, as shown in the figure in FIG. 14, the data processing unit 70 generates a reference circle having a predetermined radius centered on the designated position (point). Then, as shown in the lower figure of FIG. 14, the data processing unit 70 detects the overlap with the point data p on the laser side line, and selects a plurality of point data p included in the reference circle.

Then, as shown in FIG. 15A, the data processing unit 70 first extracts the point data p whose distance h is the maximum distance hmax from the selected plurality of point data p. It is presumed that the point data p having the maximum distance hmax is the point data p existing at the lowest position. Then, the data processing unit 70 extracts the point data p in which the distance D of the distance h is within a certain range (within 7 cm in the present embodiment) with the maximum distance hmax as a reference, and the distance h of the extracted point data p. Calculate the average value of. The data processing unit 70 estimates that the average value calculated in this way is the distance h to the ground surface F, and determines the height of the ground surface F (hereinafter referred to as the reference height H0) based on this. do.
Then, the data processing unit 70 calculates the altitude value H of the point data p from the distance h and the reference height H0. As shown in FIG. 10, the altitude value H is the height of the point data p from the reference height H0.

The guide information display device 50 is configured to generate guide information GD based on the reference height H0 of the ground surface F accurately acquired by the above processing. Therefore, the guide information display device 50 can accurately calculate the shape of the suspended load W and the features C existing around the suspended load W based on the height of the ground surface F.

Next, a case where the ground surface F is automatically estimated will be described.
In the above description, the operator specifies the ground surface F, but the guide information display device 50 may have a configuration in which the ground surface F is automatically estimated by the data processing unit 70.

When the ground surface F is automatically estimated by the data processing unit 70, as shown in FIG. 15B, the data processing unit 70 has a plurality of suspended load areas WA having the same area (160 in this embodiment). Is divided into small areas S of.

Next, the data processing unit 70 extracts the point data p having the largest distance h (the distance h is the maximum distance hmax) in each small region S, and as shown in FIG. 15 (A), the maximum distance hmax. Is used as a reference, point data p in which the distance D at the distance h is within a certain range (in the present embodiment, the distance D is within 7 cm) is extracted.

Next, the data processing unit 70 calculates the average value of the distance h of the extracted point data p in each small area S. The data processing unit 70 automatically estimates the reference height H0 of the ground surface F in each small region S from the average value of the distance h calculated in this way.

Alternatively, the data processing unit 70 further averages the average value of the distance h calculated in each small area S in all the small areas S, and automatically sets the reference height H0 of the ground surface F of the suspended load area WA from this average value. Estimate to. In this case, the data processing unit 70 uses only the small area S whose distance D from the maximum value is within a predetermined threshold value with reference to the maximum value in the average value of the distance h of each small area S. The reference height H0 is calculated.

As shown in FIG. 12, in the data processing by the data processing unit 70, "plane estimation processing" is then performed (STEP-104). The data processing unit 70 estimates the upper surfaces of the suspended load W and the feature C, which are the objects to be measured, existing in the suspended load region WA by the upper surface estimation method shown below.

When the point cloud data P for one frame is plotted on the suspended load area WA shown in the XYZ coordinate system, it is represented as shown in FIG. 13 (A). Then, when the point cloud data P in such a suspended load area WA is schematically represented, it is represented as shown in the upper figure of FIG.

The data processing unit 70 divides the point cloud data P acquired in the suspended load area WA as shown in the upper figure of FIG. 16 into layers with a predetermined thickness d in the Y-axis direction as shown in the middle figure of FIG. The point cloud data P is divided into a plurality of groups (see FIG. 13B).
At this time, the data processing unit 70 assigns an individual group ID (here, ID: 001 to 006) to each divided group, and associates each point data p with the group ID.

Then, the data processing unit 70 estimates the plane in each group by using the plurality of point data p included in the group. The "plane" referred to here is a plane existing upward in the suspended load W and the feature C, that is, the "upper surface" of the suspended load W and the feature C.

Specifically, first, as shown in FIGS. 17 and 18 above, the data processing unit 70 selects two point data p · p from a plurality of point data p · p ··· included in the same group. (Two-point selection step: STEP-201).
Then, as shown in FIGS. 17 and 18 below, the data processing unit 70 calculates the distance L1 between the two selected point data p and p (point distance calculation step: STEP-202).

Next, as shown in FIGS. 17 and 19, when the distance L1 between two points is equal to or less than a predetermined threshold value r1 (STEP-203), the data processing unit 70 has two points (2 shown by a dotted line). The two point data p · p) are considered to be on the same plane (two-point plane deemed step: STEP-204). Then, as shown in FIGS. 17 and 19 below, the data processing unit 70 calculates the center of gravity G1 of each point (here, two selected points) considered to be on the same plane (center of gravity calculation step: STEP-205). If it is determined as "no" in (STEP-203), the process returns to (STEP-201) and two new points are selected again.

Next, the data processing unit 70 searches for point data p that is a near point with respect to the calculated center of gravity G1 as shown in FIGS. 17 and 20 (neighborhood point search step: STEP-206). The "neighborhood point" referred to here is a point whose distance between points with respect to the center of gravity G1 is the threshold value r1 or less.
Then, as shown in FIGS. 17 and 20 below, if the data processing unit 70 finds the point data p that is a nearby point (STEP-207), the point data p that is a nearby point is also the two points selected earlier. It is considered to be on the same plane as the data p · p (neighborhood point plane deeming step: STEP-208).

Then, as shown in FIGS. 17 and 21 above, the data processing unit 70 returns to (STEP-205) and each point considered to be on the same plane (here, three points shown by dotted lines). A new center of gravity G2 is calculated from the point data p · p · p).

The data processing unit 70 shifts to (STEP-206) and further searches for point data p that is a neighborhood point with respect to the center of gravity G2. Then, as shown in FIGS. 17 and 21 and the lower figure of FIG. 21, the data processing unit 70 further finds the point data p as the neighborhood point (STEP-207), and the point data p as the neighborhood point is also the point data p selected earlier. It is considered that the point data p is on the same plane as (STEP-208).
Then, the data processing unit 70 searches for a nearby point while calculating a new center of gravity, and repeats the processes from (STEP-205) to (STEP-208) in order until the point data p, which is a nearby point, is no longer detected. conduct.

Then, as shown in FIG. 17, if the data processing unit 70 does not find a new neighborhood point, it determines "no" in (STEP-207), and as shown in FIG. 22, it is determined that they are on the same plane. A plane is estimated by clustering a subset (cluster) of the regarded point data p (STEP-209). The term "clustering" as used herein means that the point cloud data P, which is a set of point data p, is divided into clusters so that the point data p included in each cluster has a common feature that they are on the same plane. It is a process.
The data processing unit 70 divides the point cloud data P into point data p deemed to be on the same plane, and sets the plane cluster CL1 (see the lower figure of FIG. 16). According to the point data p belonging to the plane cluster CL1, the plane (that is, the “top surface” of the suspended load W and the feature C) can be defined. In addition, a plurality of plane clusters CL1 may exist in the group to which the same group ID is assigned.

Then, the data processing unit 70 estimates the "width" of the plane from the maximum and minimum values of the X coordinate of the point data p belonging to the plane cluster CL1, and the "depth" of the plane from the maximum and minimum values of the Z coordinate. To estimate. In this way, the data processing unit 70 defines the plane formed by the plane cluster CL1. The plane defined here may be a polygon other than a rectangle.

In such a top surface estimation method, the top surfaces of the suspended load W and the feature C can be estimated based only on the point cloud data P corresponding to the top surface acquired by the laser scanner 62. Therefore, in the top surface estimation method shown in the present embodiment, it is possible to estimate the top surface of the suspended load W and the feature C in a short time based on the point cloud data P acquired by the laser scanner 62, and by extension, the top surface of the suspended load W and the feature C can be estimated. It is possible to estimate the upper surface of the suspended load W and the feature C in real time.

Further, in such a top surface estimation method, the top surfaces of the suspended load W and the feature C can be estimated without using the statistical method, and the suspended load W and the feature C can be estimated as compared with the case where the statistical method is used. The amount of calculation required to estimate the upper surface of the above can be reduced. Therefore, in the top surface estimation method shown in the present embodiment, it is possible to estimate the top surface of the suspended load W and the feature C in a shorter time based on the point cloud data P acquired by the laser scanner 62.

In the method of estimating the upper surface of the suspended load W and the feature C shown in the present embodiment, in the crane 1, a data acquisition unit 60 is provided on the top boom member 22f of the telescopic boom 22, and the suspended load W is measured by the laser scanner 62. The case where the point group data P relating to the suspended load W, the feature C, and the ground surface F is acquired from vertically above is illustrated, but the method for estimating the upper surface of the measurement object according to the present invention is the suspended load of the crane and its surface. It is not limited to the case where the object to be measured is an object existing around the suspended load.
That is, in the top surface estimation method shown in the present embodiment, for example, a laser scanner is provided at the boom tip of a work vehicle equipped with a boom (for example, a work vehicle at a high place) or a drone, and exists vertically below the boom. It can be widely applied when acquiring point cloud data of a measurement object to be measured and estimating the upper surface of the measurement object based on the acquired point cloud data.

Next, the data processing unit 70 connects the estimated plane clusters CL1 (upper surface).
As shown in FIGS. 23 and 24 above, the data processing unit 70 selects two plane clusters CL1 and CL1 to which different group IDs are assigned from the estimated plane clusters CL1, and the data processing unit 70 selects the two plane clusters CL1 and CL1 of each plane cluster CL1. The difference dH of the altitude value H is calculated (STEP-301: elevation value difference calculation step).

Here, the data processing unit 70 searches for a combination in which the difference dH is within the threshold value r2 (STEP-302). The elevation value H of the plane cluster CL1 referred to here is an average value of the elevation values H of each point data p belonging to the plane cluster CL1.

Next, as shown in FIGS. 23 and 24, when the combination of the plane clusters CL1 in which the difference dH of the altitude value H is within the threshold value r2 is detected, the data processing unit 70 determines the plane clusters CL1. The overlap dW in the X-axis direction is detected for CL1 (STEP-303: overlap detection step). The "overlap" referred to here is the degree of overlap and the degree of separation of the planes defined by the plane cluster CL1 in the X-axis direction, and as shown in FIGS. 23 and 24, the overlap amount dW1 of the "width" is detected. In the case of (dW1> 0) or when the separation amount dW2 is equal to or less than the predetermined threshold value r3 (0 ≦ dW2 ≦ r3), “overlap” is detected.

Then, as shown in FIGS. 23 and 24, when "overlap" is detected (STEP-304), the data processing unit 70 has point data p belonging to the plane clusters CL1 and CL1 on the same plane. The two plane clusters CL1 and CL1 are combined and updated as a new plane cluster CL1 (STEP-305: plane connection step). At this time, a new elevation value H is calculated from each point data p belonging to the new plane cluster CL1.

As shown in FIG. 23, the data processing unit 70 repeats the above processing until there are no more combinations of plane clusters CL1 and CL1 satisfying the conditions (STEP-306), and creates planes existing across a plurality of groups. presume.

Then, the data processing unit 70 outputs the plane (that is, the plane cluster CL1) joined by the above joining process.
The plane defined by the plane cluster CL1 is a plane that exists upward in the suspended load W and the feature C, that is, the upper surface of the suspended load W and the feature C.

In such a plane estimation method, the plane can be estimated without using the normal vector of the point cloud data P. Therefore, there is a feature that the amount of calculation is small as compared with the case of estimating the plane using the normal vector of the point cloud data P.
Further, in such a plane estimation method, by estimating the upper surface of the suspended load W or the feature C, the suspended load W or the ground is not acquired, and the point data p of the side surface of the suspended load W or the feature C is not acquired. The three-dimensional shape of the object C can be grasped.

As shown in FIG. 12, in the data processing by the data processing unit 70, "clustering processing of the same region" is then performed (STEP-105). The term "clustering" as used herein means that the point cloud data P, which is a set of data, is divided into clusters so that the point data p included in each cluster has a common feature that they are in the "same area". It is a process.

In the "clustering process of the same area" performed here, the generated plane cluster CL1 (plane) is clustered from different viewpoints of whether or not they exist in the "same area" regardless of whether or not they form the same plane. It is a process.

Specifically, as shown in the upper figure of FIG. 25, the data processing unit 70 includes a plane cluster CL1 including point data p whose elevation value H is the maximum value Hh, and a plane cluster that is not coupled to the plane cluster CL1. Extract CL1. Then, the data processing unit 70 calculates the difference ΔH of the altitude value H of each of the extracted plane clusters CL1, and if the difference ΔH is equal to or less than a predetermined threshold value, the process proceeds to the next determination.

When moving to the next determination, the data processing unit 70 confirms the overlap in the Y-axis direction of the two plane clusters CL1 and CL1 whose difference ΔH is equal to or less than a predetermined threshold value, as shown in the middle figure of FIG.
Here, when the two plane clusters CL1 and CL1 overlap in the Y-axis direction, the data processing unit 70 sets these plane clusters CL1 and CL1 as "same region" as shown in the lower figure of FIG. 25. These plane clusters CL1 and CL1 form the same region cluster CL2.

Then, the data processing unit 70 further searches for the plane cluster CL1 including the point data p having the maximum value Hh of the altitude value H and the plane cluster CL1 uncoupled to the plane cluster CL1, and the unbound plane cluster CL1. If is extracted, the determination by the difference ΔH and the confirmation of the overlap in the Y-axis direction are performed, and if there is a plane cluster CL1 that meets the above conditions, it is further added to the same region cluster CL2.

The data processing unit 70 repeats such processing until no uncoupled plane cluster CL1 is found with respect to the plane cluster CL1 including the point data p having the maximum value Hh of the altitude value H. The data processing unit 70 forms the same region cluster CL2 by the above processing.

Then, the point data p belonging to the same region cluster CL2 formed in this way is treated as having one unit in shape in the display of the guide information GD described later, and surrounds the same region cluster CL2. The guide frame GD1 is displayed.

It is preferable that such "clustering process of the same region" is hierarchical clustering using a tree structure based on the altitude value as shown in FIGS. 26 (A) and 26 (B). The data processing unit 70 creates a tree structure for each feature C using the altitude value H in the “clustering process of the same area”. Here, the case where the hierarchical clustering using the tree structure is performed on the feature C of the first example shown in FIG. 26 (A) and the case of the feature C of the second example shown in FIG. 26 (B). The case where hierarchical clustering using a tree structure is performed is illustrated.

In the hierarchical clustering using the tree structure based on the altitude value, the data processing unit 70 sets the plane cluster CL1 having the smallest average value of the altitude value H as the “root”. Further, if there is a plane cluster CL1 having an overlap in the Y-axis direction with respect to the plane cluster CL1 constituting the "root", the data processing unit 70 extends a "branch" from the "root" and "branches". The plane cluster CL1 having the overlap is added to the tip of the "branch". Then, the data processing unit 70 sets the plane cluster CL1 having the largest average value of the altitude values H as a “child”.

Here, a method of generating the guide frame GD1 will be described.
The data processing unit 70 acquires the tree structure of the feature C created in the “clustering process of the same area”. Then, the data processing unit 70 acquires the point data p included in each plane cluster CL1 constituting the tree structure.
Next, as shown in the upper figure of FIG. 27, the data processing unit 70 acquires each point data p on the laser side line located at the innermost position in the Z-axis direction from the point data p of the “child” plane cluster CL1. do. Then, the data processing unit 70 creates a rectangle that is separated in the Z-axis direction by 1/2 of the distance from the adjacent laser lateral lines and has a width in the X-axis direction that can surround each point data p. ..

Next, when the point data p exists on the laser side line adjacent to the created rectangle, the data processing unit 70 includes all the point data p on the corresponding laser side line as shown in the lower figure of FIG. 27. Create an outline by transforming the rectangle into.
Then, the data processing unit 70 searches for the point data p on the adjacent laser side line until the point data p on the target laser side line disappears, and repeats the above processing.
Finally, the data processing unit 70 creates an outline that encloses all the plane clusters CL1 included in the selected tree structure.

Then, the data processing unit 70 outputs only the outlines that meet the conditions from the created outlines as the guide frame GD1.
As the condition for outputting as the guide frame GD1, for example, as shown in FIG. 28 (A), a condition for displaying only the outline of the outline of the feature C can be selected. When this condition is selected, the data display unit 80 displays one guide frame GD1 that surrounds the entire feature C with respect to the feature C.

Further, as a condition for outputting as the guide frame GD1, for example, as shown in FIG. 28 (B), in addition to the outline of the outline of the feature C, the difference (difference ΔH) in the altitude value H with respect to the “root” is used. , The condition for displaying the outline related to the plane cluster CL1 having the highest elevation value H in each branch can be selected from the outlines (small frames) which are equal to or more than the threshold value. When this condition is selected, the data display unit 80 displays the first guide frame GD1 that surrounds the entire feature C and the second guide frame GD1 that is included inside the first guide frame GD1. The guide frame GD1 is displayed, and more detailed guide information GD in consideration of the three-dimensional shape of the feature C is displayed.

Further, as a condition for outputting as the guide frame GD1, for example, as shown in FIG. 28 (C), in addition to the outline of the outline of the feature C, the difference (difference ΔH) in the altitude value H with respect to the “root” , It is possible to select the condition for displaying all the outlines (small frames) that are equal to or higher than the threshold value. Even when this condition is selected, the data display unit 80 displays the first guide frame GD1 that surrounds the entire feature C and the second guide frame GD1 that is included inside the guide frame GD1. More detailed guide information GD in consideration of the three-dimensional shape of the object C is displayed.

Such display conditions can also be performed by adjusting the threshold value of the difference ΔH. The operator can appropriately select the display conditions of the guide frame GD1 so that the display of the guide information GD becomes easier to see.

That is, in the guide information display device 50, by generating the guide frame GD1 based on the same region cluster CL2, the guide frame GD1 expressing the feature C in more detail is provided in consideration of the three-dimensional shape of the feature C. It becomes possible to generate. Further, the guide information display device 50 can generate a guide frame GD1 that collectively surrounds the plane clusters CL1 existing in the same region. That is, according to the guide information display device 50, it is possible to present a more detailed and easy-to-read guide information GD.

As shown in FIG. 12, in the data processing by the data processing unit 70, "synchronization processing of point cloud data and camera image" is then performed (STEP-106).
Here, as shown in FIGS. 5A and 5B, the point cloud data P acquired in the XYZ coordinate system is converted into the coordinate values in the camera space coordinate system and synchronized with the image M captured by the camera 61 ( (Alignment) and output to the data display unit 80.

As shown in FIG. 12, in the data processing by the data processing unit 70, "guide display processing" is then performed (STEP-107).
The data processing unit 70 generates guide information GD based on the generated information of the same area cluster CL2 and outputs it to the data display unit 80.
In the "guide display process", the "crane information" output from the controller 34 of the crane 1 is used. The "crane information" used here includes information related to the length of the telescopic boom 22, the undulation angle, the working radius of the crane 1, the weight of the suspended load W, and the like.

Although a series of flow of data processing by the data processing unit 70 has been described, in such a configuration, it is not necessary to acquire the point data p on the side surface of the object to be measured, and the suspended load W and the feature C can be calculated with a small amount of calculation. The guide information GD can be generated by accurately grasping the three-dimensional shape of. Since such a configuration requires a small amount of data calculation, it is suitable for the purpose of grasping the shapes of the suspended load W and the feature C in real time, and the data processing unit 70 having a simple hardware configuration can be used. ..

Next, the contents of the guide information GD will be described.
In the guide information display device 50, the guide information GD is displayed by the data display unit 80. The guide information GD displayed by the data display unit 80 includes information related to a position designated by the operator on the ground surface F as shown in FIG. 8 (B).

Further, in the guide information display device 50, the suspended load W can be specified. Similar to the case where the operator designates the ground surface F, by instructing the suspended load W on the screen, the plane (upper surface) existing at the designated position is set as representing the upper surface of the suspended load W. After being designated as the suspended load W, it is preferable that the guide frame GD1 related to the suspended load W and the guide frame GD1 related to the feature C are displayed separately by changing the line color, line thickness, and the like.
Information relating to the designated positions of the ground surface F and the suspended load W is displayed by markers represented by figures such as circles.

Further, the guide information GD displayed by the data display unit 80 includes the guide frame GD1 generated by the data processing unit 70.

The data processing unit 70 outputs the guide frame GD1 based on the set same area cluster CL2. The data processing unit 70 can provide a margin for the guide frame GD1 of the suspended load W to surely avoid a collision, and the frame is offset outward by a predetermined distance from the outer line of the suspended load W. The line can be output as the guide frame GD1. Such a guide frame GD1 has a frame display in which the upper surface (plane cluster CL1) estimated in the suspended load W and the feature C is surrounded by a line segment.

Further, the guide information GD displayed by the data display unit 80 includes height information GD2 from the reference height H0 to the lower surface of the suspended load W and height information GD3 from the reference height H0 to the upper surface of the feature C. include.

It is preferable that the height information GD2 of the suspended load W is configured to provide an independent area at an easily visible position on the screen of the data display unit 80 and display the data in that area.
In the guide information display device 50, the height information GD2 of the suspended load W and the height information GD3 of the feature C are not mistaken for each other by such a configuration.

The data processing unit 70 calculates the height information GD2 by subtracting the height of the suspended load W from the height of the upper surface of the plane cluster CL1 estimated to be the upper surface of the suspended load W.
In the guide information display device 50, the operator inputs information related to the suspended load W (hereinafter referred to as “suspended load information”) to the data processing unit 70 in advance. The input of "suspended load information" by this operator is performed from the data input unit 90. Then, the data processing unit 70 acquires the height of the suspended load W by using the “suspended load information”.

The guide information display device 50 is configured to display the height information GD3 of the feature C inside the guide frame GD1 surrounding the feature C. Alternatively, in the guide information display device 50, when the guide frame GD1 is small, the guide frame GD1 is displayed so as to partially overlap the guide frame GD1.
In the guide information display device 50, the correspondence between the feature C and the height information GD3 is clarified by such a configuration.

Further, in the guide information display device 50, the data processing unit 70 is configured to change the line color of the guide frame GD1 according to the altitude value H of the plane cluster CL1 corresponding to the guide frame GD1.
With such a configuration, in the guide information display device 50, the operator can perceive the rough altitude value (height) of the suspended load W and the feature C sensuously by looking at the guide frame GD1. Therefore, the guide information display device 50 can more accurately present the heights of the suspended load W and the feature C.

Further, the guide information display device 50 is configured such that the data processing unit 70 changes the font color of the height information GD2 according to the altitude value H of the plane cluster CL1 corresponding to the guide frame GD1.
With such a configuration, in the guide information display device 50, the operator can perceive the approximate altitude value (height) of the suspended load W or the feature C sensuously by looking at the height information GD2. Therefore, the guide information display device 50 can more accurately present the heights of the suspended load W and the feature C.

Further, the display of the guide information GD performed by the guide information display device 50 includes the flow line information of the suspended load W. The flow line information of the suspended load W includes the working radius information GD4 of the suspended load W and the axis line information GD5 of the telescopic boom 22 of the crane 1.

The working radius information GD4 serves as a guideline for the flow line of the suspended load W when the telescopic boom 22 is swiveled from the present state, and the suspended load W moves along the arc shown as the working radius information GD4.

Further, the axis line information GD5 serves as a guideline for the flow line of the suspended load W when the telescopic boom 22 is undulated and expanded / contracted from the current state, and the suspended load W moves along a straight line shown as the working radius information GD4. do.

The guide information display device 50 generates the working radius information GD4 of the suspended load W and the axis line information GD5 of the telescopic boom 22 based on the “crane information”.
The data processing unit 70 calculates the working radius of the crane 1 based on the "crane information", generates an arc indicating the working radius, and outputs the working radius information GD4.
Further, the data processing unit 70 calculates the axial direction of the telescopic boom 22 based on the "crane information", generates a straight line indicating the axial direction, and outputs it as the axis information GD5.

Further, in the guide information display device 50, the line displaying the working radius information GD4 and the axis line information GD5 is represented by a broken line, and the length and interval of the broken line are used as a guide (hereinafter referred to as a reference length). It is configured to be displayed with. For example, when the reference length is 1 m, the working radius information GD4 and the axis line information GD5 change the length and interval of the broken lines on the display according to the size of the suspended load area WA displayed on the data display unit 80. However, on the scale at that time, it is displayed as a length and an interval corresponding to 1 m on the ground surface F.
In the guide information display device 50, by displaying the length and interval of the broken lines with a reference length (for example, 1 m), the operator can feel the scale feeling of the suspended load W and the feature C from the guide information GD. It is said.

Further, the data processing unit 70 calculates the height of the data acquisition unit 60 and the size of the suspended load area WA and the size of the display range of the data display unit 80 based on the “crane information”. The scale of the broken line (the size of the broken line and its interval) displayed as the working radius information GD4 and the axis line information GD5 is changed according to the calculation result.

Further, the display of the guide information GD performed by the guide information display device 50 includes an alarm display for preventing contact between the suspended load W and the feature C.
In the data processing unit 70, the horizontal distance when the suspended load W and the feature C are projected onto the horizontal plane is equal to or less than a predetermined threshold value (for example, 1 m), and the vertical distance is a predetermined threshold value (for example, 1 m). ) If it is less than the above, it is judged that there is a risk of contact.

At this time, as shown in FIG. 29, the data processing unit 70 emphasizes the guide frame GD1 and the height information GD2 of the feature C which may come into contact with the suspended load W, and the guide frame GD1 and the height information GD2 of the feature C are emphasized. Output height information GD2. Alternatively, the data processing unit 70 outputs the guide frame GD1 and the height information GD2 of the feature C in a manner of blinking the guide frame GD1 and the height information GD2 of the feature C. In the guide information display device 50, the data processing unit 70 outputs the guide frame GD1 and the height information GD2 of the feature C, which is an alarm display, and displays them on the data display unit 80 to call the operator's attention. ..

Further, the guide information display device 50 is configured to exclude the point cloud data P existing between the suspended load W and the top boom member 22f when the data processing unit 70 generates the guide information GD. A process of excluding the point cloud data P existing between the suspended load W and the top boom member 22f will be described with reference to FIGS. 30 and 31.

The data processing unit 70 determines whether or not there is an outline created for the suspended load W (STEP-401). The data processing unit 70 acquires all the plane clusters CL1 (plane cluster acquisition step: STEP-402) when there is an outline created for the suspended load W, and there is no outline created for the suspended load W. , The process of excluding the point cloud data P existing between the suspended load W and the top boom member 22f is completed.

Then, the data processing unit 70 acquires the outline of the plane cluster CL1 having the highest elevation value H (outline acquisition step: STEP-403).

The data processing unit 70 calculates the center of gravity of the acquired outline, and sets this center of gravity as the suspension position WP of the suspended load W (see FIGS. 4 and 31 (A)) (STEP-404).

The data processing unit 70 repeats (STEP-405) to (STEP-411) in order as many as the number of acquired plane clusters CL1. First, the data processing unit 70 determines whether or not the altitude value H of the selected plane cluster CL1 is higher than the altitude value H of the height information GD2 of the suspended load W (see FIG. 31 (A)) (STEP-405). ). When the data processing unit 70 determines that the altitude value H of the selected plane cluster CL1 is higher than the altitude value H of the height information GD2 of the suspended load W, the suspension position WP of the suspended load W and the center of gravity G3 of the suspended load W Calculate a straight line connecting the two (see FIG. 31 (A)) (straight line calculation step: STEP-406), and determine that the elevation value H of the plane cluster CL1 is not higher than the elevation value H of the height information GD2 of the suspended load W. If so, the selected plane cluster CL1 is not excluded.

Then, the data processing unit 70 calculates the horizontal distance HD between the center of gravity G3 of the selected plane cluster CL1 and the calculated straight line (see FIG. 31 (A)) (horizontal distance calculation step: STEP-407).

The data processing unit 70 has a suspended load width WD from the position of the center of gravity of the suspended load W (center of gravity G3) to the outer circumference of the suspended load W (outline line of the suspended load W) that is farthest from the position of the center of gravity of the suspended load W in the calculated plan view from the linear direction. (See FIG. 31 (B)), and it is determined whether or not the calculated horizontal distance HD is smaller than the suspended load width WD (STEP-408). When the data processing unit 70 determines that the calculated horizontal distance is smaller than the suspended load width WD, the data processing unit 70 excludes the point cloud data P forming the selected plane cluster CL1 (plane cluster exclusion step: STEP-411). When calculating the suspended load width WD, as described above, it may be calculated based on the acquired point cloud data P, or the "suspended load information" (suspended load W) input by the operator. Width and depth) may be used for calculation.

When the data processing unit 70 determines that the horizontal distance is not smaller than the suspended load width WD, the selected plane cluster CL1 overlaps the suspended load W in the Yc-axis direction in the camera space coordinate system (not shown in FIG. 31). It is determined whether or not it is (STEP-409). When the data processing unit 70 determines that the selected plane cluster CL1 overlaps the suspended load W in the Yc-axis direction view in the camera space coordinate system, the data processing unit 70 excludes the point group data P forming the selected plane cluster CL1 ( STEP-411).

When the data processing unit 70 determines that the selected plane cluster CL1 does not overlap the suspended load W in the Yc axis direction view in the camera space coordinate system, the selected plane cluster CL1 is in the XYZ coordinate system (see FIG. 31 (A)). ), It is determined whether or not it overlaps with the suspended load W in the Y-axis direction (STEP-410). When the data processing unit 70 determines that the selected plane cluster CL1 overlaps the suspended load W in the Y-axis direction in the XYZ coordinate system, the data processing unit 70 excludes the point group data P forming the selected plane cluster CL1 (STEP). -411) If it is determined in the XYZ coordinate system that the selected plane cluster CL1 does not overlap with the suspended load W in the Y-axis direction, the selected plane cluster CL1 is not excluded.

The point cloud data P excluded in this way includes the point cloud data P corresponding to the main wire rope 27. As a result, the guide information display device 50 accurately presents information on the shape and position of the suspended load W and the feature C existing around the suspended load W regardless of the posture and operating condition of the crane 1. Can be done. Further, the guide frame GD1 is generated by excluding the point cloud data P corresponding to the main wire rope 27, and the guide information display device 50 can present the guide information GD that is more accurate and easy to see.

Further, as shown in FIG. 32, as another embodiment, an exclusion region JA excluding the point cloud data P may be set between the suspended load W and the top boom member 22f. The exclusion area JA is set within an arbitrary range from the straight line connecting the suspension position WP of the suspended load W and the center of gravity of the suspended load W. In the present embodiment, the set range is set so that the upper surface of the suspended load W is the lower end. The height of the lower end of the exclusion area JA should be set at a position separated from the upper surface of the suspended load W by a predetermined distance in consideration of not affecting the generation of the guide frame GD1 of the suspended load W. Is preferable.

The point cloud data P acquired in the exclusion area JA includes the point cloud data P corresponding to the main wire rope 27. As a result, the guide information display device 50 accurately presents information on the shape and position of the suspended load W and the feature C existing around the suspended load W regardless of the posture and operating condition of the crane 1. Can be done. Further, the guide frame GD1 is generated by excluding the point cloud data P corresponding to the main wire rope 27, and the guide information display device 50 can present the guide information GD that is more accurate and easy to see.

In the guide information display device 50 having such a configuration, the guide frame GD1 indicating the shape and the height of the suspended load W and the feature C existing around the suspended load W are indicated to the operator of the crane 1. The guide information GD including the height information GD2 and GD3 can be accurately presented. Then, if the guide information display device 50 having such a configuration is used, for example, even in a situation where the operator cannot directly visually recognize the suspended load W, the operator can display the guide information GD presented by the guide information display device 50. Based on this, the work by the crane 1 can be performed efficiently and safely.

1 Crane 50 Guide information display device 60 Data acquisition unit 61 Camera 62 Laser scanner 70 Data processing unit 80 Data display unit P Point group data W Suspended load F Ground surface C Feature M (photographed by camera) Video GD guide information GD1 guide Frame GD2 (lower surface of suspended load) Height information GD3 (upper surface of feature) Height information WA Suspended load area CL1 Flat cluster CL2 Same area cluster HD Horizontal distance WD Suspended load width WP Suspended position JA Excluded area

Claims (5)

A crane guide information display device including a camera, a laser scanner, a data processing unit, and a data display unit.
The camera captures a part of the working area of the crane.
The laser scanner acquires point cloud data of the work area taken by the camera, and obtains point cloud data.
The data processing unit excludes the point group data existing between the suspended load suspended by the crane and the tip of the boom of the crane from the point group data acquired by the laser scanner, and the remaining point group data. Based on the point group data, the upper surface of the suspended load, the ground surface of the working area, and the upper surface of the feature existing in the working area are estimated, and the upper surface of the suspended load and the upper surface of the feature are surrounded, respectively. A guide information display device that generates a guide frame image, superimposes the guide frame image generated by the data processing unit on an image captured by the camera, and displays the guide frame image on the data display unit. The data processing unit
A plane cluster that estimates that point cloud data within a predetermined elevation difference from an arbitrary elevation value is on the same plane between the suspended load suspended by the crane and the tip of the boom of the crane. Form and
A straight line connecting the center of gravity of the suspended load and the suspended position of the suspended load at the tip of the boom is calculated.
The suspended load width from the position of the center of gravity of the suspended load to the outer circumference of the suspended load, which is the farthest from the position of the center of gravity of the suspended load in the plan view from the linear direction, is calculated.
Calculate the horizontal distance between the center of gravity of the plane cluster and the straight line,
When the horizontal distance is smaller than the suspended load width, or when the suspended load and the planar cluster overlap in a plan view from the vertical direction, the point group data forming the planar cluster is collected by the laser scanner. The guide information display device according to claim 1, wherein the point group data is excluded from the acquired point group data. The data processing unit
An exclusion area is set between the suspended load and the tip of the boom.
The guide information display device according to claim 1, wherein the point cloud data acquired in the exclusion region is used as point cloud data to be excluded from the point cloud data acquired by the laser scanner. A crane provided with the guide information display device according to any one of claims 1 to 3. A camera that captures a part of the crane's work area,
In the work area, a laser scanner that acquires point cloud data from above the suspended load, and
A data acquisition unit that acquires point cloud data of the work area taken by the camera by the laser scanner, and a data acquisition unit.
Based on the point cloud data, the upper surface of the suspended load, the ground surface of the working area, and the upper surface of the feature existing in the working area are estimated, and the upper surface of the suspended load and the upper surface of the feature are respectively. A data processing unit that generates a guide frame image that surrounds it,
A data display unit that superimposes and displays the guide frame image generated by the data processing unit on the image captured by the camera, and a data display unit.
It is a guide information display method in a guide information display device provided with.
A plane cluster acquisition step of acquiring a plane cluster formed by estimating that the point cloud data within a predetermined altitude difference from an arbitrary altitude value is on the same plane, and
The outline acquisition step of acquiring the outline having the highest elevation value in the outline of the plane cluster, and
A suspension position calculation step of calculating by regarding the center of gravity of the outline as the suspension position of the suspended load at the tip of the boom of the crane.
A straight line calculation step of calculating a straight line connecting the center of gravity of the suspended load to the suspended position, and
A horizontal distance calculation step for calculating the horizontal distance between the center of gravity of the plane cluster and the straight line, and
When the horizontal distance is smaller than the suspended load width, which is the distance from the center of gravity of the suspended load to the outer periphery of the suspended load in the plan view from the linear direction, or in the plan view from the vertical direction. A guide information display method comprising a plane cluster exclusion step of excluding the point group data forming the plane cluster from the point group data acquired by the laser scanner when the suspended load and the plane cluster overlap.

Images