JP2968207B2 - Departure monitoring method for heavy equipment for remote construction - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for monitoring deviation of a heavy construction equipment for remote construction, and more particularly to monitoring of a heavy construction equipment for remote construction in a construction area for remote construction deviating from a safe work area in the construction area by a stereo image pair. On how to do it.

[0002]

2. Description of the Related Art When constructing a dam in an area where people are restricted or an area where there is danger to humans, the construction area is made unmanned and construction work machines (hereinafter referred to as heavy machinery) such as vibrating rollers, bulldozers, and dump trucks. ) May be made semi-automatic or remotely operable for remote construction. FIG. 5 shows an example of such remote construction. In FIG. 5, a plurality of remotely controllable heavy machines 3a are arranged in the construction area 1, an image pickup device 16 and an image transmitter 17 are attached to each heavy device 3a, and the output image of each image pickup device 16 is the receiving station 4. The image is transmitted to the image receiver 13 in the remote control room 40, and the operator in the remote control room 40 views each transmission while viewing the transmission image.
Remotely controlling 3a. Further, a position measuring device 9 using a satellite navigation system (hereinafter referred to as GPS) is attached to each heavy equipment 3a, and three-dimensional coordinates of each heavy equipment 3a in the construction area 1 are grasped in the remote control room 40 based on the position signals. I have. Reference numeral 6 in the figure
Denotes an imager for monitoring and / or surveying the construction area 1, an image transmitter 7 for transmitting an output image thereof to the receiving station 4,
Reference numeral 5 denotes a photographing moving body. Reference numeral 41 denotes a data transmission tower, and reference numeral 43 denotes a survey helicopter.

[0005] In remote construction as shown in FIG. 5, deviation of a heavy machine from a work area due to runaway of a semi-automatic heavy machine or a mistake in remote operation or fall from an embankment (hereinafter, collectively referred to as a deviation from a safe work area). ) Heavy machinery to avoid accidents
3a needs to be monitored. This is because it is extremely difficult to recover heavy equipment that deviates from the safe work area in remote construction where the construction area is unmanned. Conventionally, as one method of monitoring the deviation of a heavy machine from a safe work area in remote construction, a plurality of monitoring imagers 6 are provided in the construction area 1 and the heavy machine 3a is measured from a monitoring image of each imager 6 by a stereo image measurement method. A method has been proposed in which the three-dimensional coordinates of the heavy equipment 3a are compared with the three-dimensional design drawings of the construction area 1 created in advance by CAD or the like.

As shown in FIG. 7, the stereo image measurement method is based on the two-dimensional coordinates of an image of the object 2 on a pair of images photographed from different directions by the imaging devices 6a and 6b at different positions. Is a measurement method for obtaining Referring to FIG. 7 and equations (1) to (11), the three-dimensional coordinates of the point P on the target 2 in the earth coordinate system are (X, Y, Z), and the point P in the image coordinate system of the imaging device 6a is image
Assuming that the two-dimensional coordinates of Pa are (Xa, Ya) and the two-dimensional coordinates of the image Pb of the point P in the image coordinate system of the imaging device 6b are (Xb, Yb), the conversion formula from the point P to the image Pa and the image Pb Can be represented by equations (1) and (2). Here, the matrices A and B indicate conversion parameters determined according to the position / posture of the imaging device, the focal length of the lens, and the like.
(X, Y, Z, 1) ^T , (Xa, Ya, 1) ^T and (Xb, Yb, 1) ^T represent the coordinates of point P, image Pa, and image Pb in a homogeneous coordinate system representation. T indicates a transposed matrix). The coordinates (Xa, Ya) and (Xb, Yb) are obtained from the images of the respective imaging devices 6a and 6b, and are substituted into the equations (1) and (2).
Formulas (1) and (2) are expanded using Ha and Hb expressed by formulas (3) and (4).
Equations (5) to (8) are obtained by rearranging. Formulas (5) to (8) are summarized using matrices F, Q, and V defined in formula (9), and F = QV
(Equation (10)), and assuming that the inverse matrix Q ⁻¹ exists, the equation (1)
The coordinates (X, Y, Z) can be calculated by 1). In short, if the conversion parameters A and B are determined, the three-dimensional position of the object 2 can be calculated from the two-dimensional coordinates of the object 2 on the image pair by using equation (11). Note that the number of rows of the matrices F and Q increases according to the number of imaging devices.

[0005]

(Equation 1)

The present inventors have developed a method for accurately identifying the above-mentioned conversion parameters A and B in an unmanned construction area, and disclosed it in Japanese Patent Application Laid-Open No. Hei 8-101035 as a remote type surveying method.
Referring to FIG. 6 and equations (12) to (18), the invention disclosed in the publication will be briefly described. Image sensors 6a and 6b with an image transmitter 7 are provided facing the construction area 1, and a GPS position measuring device is provided. 9 and the radio transmitter 10 is mounted remotely operable mobile 3b the imager 6a, is moved to the reference point P ₁ in the field of view of 6b, the three-dimensional coordinates of the reference points P ₁ measured by the position measuring device 9 ( X ₁ , Y ₁ , and Z ₁ ) are transmitted from the wireless transmitter 10. The radio receiver 14 of the receiving station 4 receives the three-dimensional coordinates (X ₁ , Y ₁ , Z ₁ ) of the reference point P ₁ , and further receives the image receiver 13 of the receiving station 4.
The moving body 3b on the output images of the imaging devices 6a and 6b received at
Two-dimensional coordinates P _a1 = (X _a1 , Y _a1 ) and P _b1 = (X _b1 , Y _b1 )
Detecting the three-dimensional coordinates P ₁ and the two-dimensional coordinates P _a1 and the formula (5) (6)
Into equations (12) and (13). Obtains a formula corresponding to Formula (12) (13) at the reference point P ₁ to P ₆ having different at least 6, the formula
By using the matrices X, Y, and R defined in (16) and setting A ₃₄ = 1, the matrix Y, that is, the matrix A can be identified as in equation (18). However, the reference points P _{1 to} P ₆ must not all be on the same plane. For matrix B, coordinates P ₁
And coordinates P _b1 are substituted into equations (7) and (8) to obtain equations (14) and (15),
By obtaining a formula corresponding to Formula (14) (15) at each reference point P ₂ to P _6, it can be identified as in the case of the matrix A.

[0007]

(Equation 2)

[0008]

According to the above-mentioned stereo image measuring method, the three-dimensional coordinates of the heavy equipment 3 in the construction area 1 can be grasped. Therefore, it is difficult to monitor the heavy equipment 3 only by comparing the three-dimensional design drawing created in advance with the three-dimensional position of the heavy equipment 3. That is, in order to monitor the deviation of the heavy equipment 3 from the safe work area, the safety work area in the construction area 1 that changes according to the progress of the construction is grasped at any time, and the safe work area and the three-dimensional coordinates of the heavy equipment 3 are determined. It is necessary to compare In the remote construction, the safe work area must also be grasped based on the monitoring image or the like. However, in order to compare with the three-dimensional coordinates of the heavy equipment 3, the three-dimensional coordinates of the safe work area are obtained from the two-dimensional monitoring image. There is a need.

It is an object of the present invention to provide a method for monitoring deviation of a heavy construction equipment for remote construction from a safe work area by using stereo images.

[0010]

Referring to FIGS. 1 and 2, a method for monitoring deviation of a remote construction heavy machine according to the present invention comprises a construction heavy machine 3 operated in a construction area 1 for remote construction. An image transmitter for capturing a stereo image pair Ia, Ib in a method for monitoring departure from a safe working area 1a in an area 1
The imager pair 6a, 6b with 7a, 7b is exposed to the construction area 1; the image receivers 13a, 13b received by the receiving station 4 from the image transmitters 7a, 7b.
And monitors 15a and 15b for displaying the image pairs Ia and Ib; two-dimensional coordinates on the image pairs Ia and Ib with respect to a plurality of reference points whose three-dimensional coordinates in the construction area 1 are known; Conversion of stereo image measurement method for calculating three-dimensional coordinates in construction area 1 from two-dimensional coordinates of arbitrary points on image pair Ia, Ib based on comparison of three-dimensional coordinates and two-dimensional coordinates on image pair Ia, Ib Parameter
Define 26; pen-input limit lines 27a, 27b corresponding to the boundaries of the safe work area 1a on the image pair Ia, Ib (see FIG. 3) and two-dimensional limit lines 27a, 27b on the image pair Ia, Ib. From the coordinates and the conversion parameters 26, the limit lines 27a, 2
Calculate and store the three-dimensional coordinates 28 of 7b; detect the two-dimensional coordinates of the image of the heavy equipment 3 on the image pair Ia, Ib, and calculate the three-dimensional coordinates of the heavy equipment 3 based on the two-dimensional coordinates of the heavy equipment 3 and the conversion parameter 26. Calculation, calculation of the three-dimensional distance between the three-dimensional coordinates 28 of the limit line 27 and the three-dimensional coordinates of the heavy equipment 3, and comparison of the three-dimensional distance with the predetermined lower limit of the three-dimensional distance allow the safe work area of the heavy equipment 3. It monitors the deviation from 1a.

[0011]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 1 shows an embodiment in which a heavy equipment 3 for remote construction operated on the top of an embankment constructed in a construction area 1 monitors the fall from the top of the embankment by the method of the present invention. That is, in the same figure, the top surface of the embankment is the safe work area 1a of the heavy equipment 3. An imager pair 6a, 6b is provided facing the construction area 1, and an imager pair 6a, 6b
Are transmitted from the image transmitters 7a and 7b to the image receivers 13a and 13b of the receiving station 4. One example of the image pickup device 6 is a CCD camera, and one example of the image transmitter 7 and the image receiver 13 is a CCD camera.
It is a simple image radio in the GHz band. An image processing device 20 is provided in the receiving station 4, and the image pair Ia, received by the image receivers 13 a and 13 b,
Ib is input to the image processing device 20, and the frame memories 21a, 21b
And the contents of the frame memories 21a and 21b are displayed on the monitors 15a and 15b of the image processing device 20. Although the optical axes of the imaging device pairs 6a and 6b are orthogonal to each other in FIG.
The optical axes are not limited to orthogonal directions as long as the directions are different from each other. Although FIG. 1 shows a case where two imagers 6 are used, three or more imagers 6 may be used.

FIG. 2 shows an example of a flow chart of the monitoring method of the present invention. Referring to the figure, first, in step 201, a conversion parameter 26 of the stereo image measurement method is determined. The conversion parameters include the matrices A, B, and the like in the above equations (1) and (2). For example, in the construction area 1, three or more reference points whose three-dimensional coordinates are known are determined, the two-dimensional coordinates on the image pair Ia, Ib for each reference point are determined, and the three-dimensional coordinates and two-dimensional coordinates of each reference point are determined. Formula (16) ~
From (18), the matrices A and B can be determined. In FIG. 1, the heavy equipment 3 on which the GPS position measuring device 9 and the wireless transmitter 10 are mounted is moved to a reference point in the field of view of the imager pair 6a, 6b, and the measurement signal of the position measuring device 9 is transmitted from the wireless transmitter 10. The three-dimensional coordinates of the reference point transmitted and received by the wireless receiver 14 of the receiving station 4 are input to the calculating means 24 of the image processing device 20. The two-dimensional coordinates of the image of the heavy equipment 3 are detected from the image pair Ia and Ib in the frame memory 21 by the coordinate detecting means 22 of the image processing device 20, and the detected two-dimensional coordinates are input to the calculating means 24. The calculating means 24 converts the conversion parameters based on the three-dimensional coordinates inputted from the wireless receiver 14 and the two-dimensional coordinates inputted from the coordinate detecting means 22.
Define 26. In order to facilitate the detection of the image of the heavy equipment 3 by the coordinate detecting means 22, an optotype 11 (see FIG. 6) of a predetermined color may be attached to the heavy equipment 3.

Next, in step 202, the stereo image pair Ia, Ib
Enter limit line 27 above. Here, the limit line is a safe work area 1a of the heavy equipment 3 by an emergency stop signal or the like.
Means the limit that deviation from the safe work area can be avoided.
Are assumed to be separated by a predetermined distance from the inside of the boundary. FIGS. 3A and 3B show an example of a method of inputting a limit line on the pair of images Ia and Ib displayed on the monitors 15a and 15b, and are inward by a predetermined distance L from a boundary on the pair of images Ia and Ib. Limit lines 27a and 27b are input by light pens 16a and 16b. The predetermined distance L is a distance determined based on a scale of the length on the monitor 15 with respect to the brake distance of the heavy equipment 3 and the distance of the construction area at the time of an emergency stop. Light pens 16a, 16b
Is input to the coordinate detecting means 22 via the frame memories 21a and 21b, and the two-dimensional coordinates of each point on the limit line 27 are detected. For example, input data from the light pen 16 can be displayed on a monitor as a white line, and the center axis of the white line can be detected as the two-dimensional coordinates of the limit line.

In step 203, three-dimensional coordinates 28 are calculated from the two-dimensional coordinates 27 of the limit line. For example, step 20
The three-dimensional coordinates 28 of the limit line can be obtained as a straight line or a line segment on the ground surface in the construction area 1 based on the two-dimensional coordinates 27, the conversion parameters 26, and the equation (11) obtained in Step 2. Alternatively, the limit line may be obtained as three-dimensional coordinates of a vertical plane including the straight line or the line segment. Calculation of the three-dimensional coordinates 28 of the limit line is performed by the calculating means 24, and the calculation result is stored in the calculating means 24.

After calculating the three-dimensional coordinates 28 of the limit line,
The two-dimensional coordinates of the image of the heavy equipment 3 on the image pair Ia, Ib are detected by the coordinate detecting means 22 (step 204), and the calculating means 24
To calculate the three-dimensional coordinates of the heavy equipment 3 and the three-dimensional distance between the limit line 27 and the heavy equipment 3 (steps 205 to 20).
6). In step 207, it is determined whether the heavy equipment 3 is inside the limit line 27. If not, the heavy equipment 3 is stopped urgently (step 208). The distance is compared with a predetermined lower limit of the three-dimensional distance (step 209). An example of the predetermined lower limit is a brake distance b _Lim with respect to the maximum speed V _Lim of the heavy equipment 3 (see FIG. 4). In the flowchart of FIG. 2, when the three-dimensional distance is equal to or less than a predetermined lower limit, a speed limit for the heavy equipment 3 is set (step 210), and when the three-dimensional distance is larger than the predetermined lower limit, the set speed limit is released. (Step 211). Here, the speed limit means an upper limit speed of the heavy equipment 3 allowed by the operation of the remote operator. When releasing the speed limit, set the speed of heavy equipment 3 to the maximum speed V
It is allowed to increase to _Lim, but when setting the speed limit, the speed of the heavy equipment 3 is prohibited to be equal to or higher than the set upper limit speed.

By repeating steps 202 to 211, it is possible to prevent the heavy machine 3 from deviating from the safe work area in the construction area. When monitoring based on the same limit line as the previous cycle is performed in each cycle except the first cycle, steps 202 to 203 can be omitted. Step 20
By resetting the limit line in 2 to 203, monitoring based on the new limit line can be continued. That is, the safe work area can be set again in the construction area according to the progress of the construction.

Thus, the object of the present invention, that is, the method of monitoring the deviation of the heavy construction equipment for remote construction from the safe working area by the stereo image can be provided.

[0018]

In the embodiment shown in FIG. 1, the three-dimensional coordinates of the heavy equipment 3 and the three-dimensional distance between the limit line 27 and the heavy equipment 3 obtained by the image processing means 20 are output to the computer 30. To 211 are processed by the computer 30. The relational expression 31 between the speed V _{0 of the} heavy equipment 3 and the brake distance b is stored in the computer 30 of FIG. The relational expression 31 is measured and stored in advance. FIG. 4 shows an example of the relational expression 31. Computer 30 of FIG. 1, the brake distance calculates the actual velocity V _t of heavy machinery 3 from temporal changes in the three-dimensional coordinates of the heavy equipment 3 inputted from the image processing unit 20, corresponding to the relationship equation 31 and the actual velocity V _t b _t is obtained, and the obtained brake distance b _t is calculated in step 20 in FIG.
The deviation of the heavy equipment 3 is monitored as the predetermined lower limit of 9. That is, since the upper limit speed of the heavy equipment 3 is limited as it approaches the limit line, it is possible to reliably prevent the heavy equipment 3 from deviating from the safe work area.

The monitoring method when one heavy machine 3 is used has been described above. However, according to the present invention, a plurality of heavy machines 3 arranged in the construction area 1 can be monitored simultaneously. In this case, a target of different color is attached to each heavy equipment,
Each heavy machine may be identified by the coordinate detecting means 22 of the image processing device 20. Further, a different limit line can be set for each heavy equipment 3, and a deviation from a safe work area different for each heavy equipment 3 can be monitored.

[0020]

As described above, the deviation monitoring method of the heavy equipment for remote construction according to the present invention provides the three-dimensional coordinates of the limit line set on the stereo image pair of the construction area and the image of the heavy equipment on the stereo image pair. Since the deviation of the heavy equipment from the safe working area is monitored based on the three-dimensional distance from the three-dimensional coordinates obtained from the two-dimensional coordinates, the following remarkable effects are obtained.

(A) The deviation of the heavy equipment for remote construction from the safe work area in the construction area which changes according to the progress can be surely monitored. (B) Obtain the relational expression between the speed of heavy equipment and the braking distance in advance,
The allowable upper limit speed of the heavy equipment can be set according to the three-dimensional distance between the heavy equipment and the limit line. (C) Different limit lines can be set for each of a plurality of heavy machines arranged in the construction area, and deviations from the safe work area different for each heavy machine can be monitored.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram of one embodiment of the present invention.

FIG. 2 is an example of a flowchart of the present invention.

FIG. 3 is an explanatory diagram of a limit line input method.

FIG. 4 is an explanatory diagram of a relational expression between a speed of a heavy machine and a brake distance.

FIG. 5 is an explanatory diagram of an example of a construction area by remote construction.

FIG. 6 is an explanatory diagram of a parameter identification method of a stereo image measurement method.

FIG. 7 is an explanatory diagram of the principle of the stereo image measurement method.

[Explanation of symbols]

1 ... Construction area, 1a ... Safe work area, 2 ... Target,
3 heavy equipment, 3a moving object, 4 receiving station, 5 moving object for shooting, 6 image pickup device, 7 image transmitter, 9 GPS position measuring device, 10 radio transmitter,
11: Target, 13: Image receiver, 14: Wireless transmitter, 15: Monitor, 16: Imager, 17: Image transmitter, 20: Image processor, 21: Frame memory,
22 ... Coordinate detection means 24 ... Calculation means 26 ... Conversion parameters 27 ... Two-dimensional coordinates of limit line 28 ... Three-dimensional coordinates of limit line 30 ... Computer 31. Relational expression.

Continuation of front page (56) References JP-A-3-39525 (JP, A) JP-A-61-62905 (JP, A) JP-A-3-62906 (JP, A) JP-A-59-98213 (JP, A) JP-A-6-28031 (JP, A) JP-A-59-106634 (JP, A) (58) Fields studied (Int. Cl. ⁶ , DB name) G05D 1/00 G05D 1/02

Claims (6)

(57) [Claims] In a method for monitoring that a heavy construction machine operated in a construction area for remote construction deviates from a safe work area in the construction area, a construction is provided for constructing a pair of imagers with an image transmitter for photographing a stereo image pair. A receiving station is provided with an image receiver for receiving from the image transmitter and a monitor for displaying the image pair; two-dimensional coordinates on the image pair corresponding to a plurality of reference points whose three-dimensional coordinates are known in the construction area; Stereo image measurement for obtaining three-dimensional coordinates and calculating three-dimensional coordinates in a construction area from two-dimensional coordinates of an arbitrary point on the image pair based on a comparison between the three-dimensional coordinates of each reference point and the two-dimensional coordinates on the image pair A limit line corresponding to the boundary of the work area is input with a pen on the image pair, and the limit line is determined from the two-dimensional coordinates of the limit line on the image pair and the conversion parameter. And calculates and stores the three-dimensional coordinates of; detection of two-dimensional coordinates of the image of the heavy equipment on the image pair,
Calculation of three-dimensional coordinates of the heavy equipment based on the two-dimensional coordinates of the heavy equipment and the conversion parameter, calculation of a three-dimensional distance between the limit line and the heavy equipment, and a predetermined lower limit of the three-dimensional distance and the three-dimensional distance A deviation monitoring method for heavy equipment for remote construction, wherein the deviation of the heavy equipment from the safe work area is monitored by comparison with the above. 2. The monitoring method according to claim 1, wherein a relational expression between the speed of the heavy equipment and the brake distance is measured and stored, and the speed of the heavy equipment is calculated from a change over time of three-dimensional coordinates of the heavy equipment; A deviation monitoring method for a heavy equipment for remote construction, wherein a brake distance corresponding to the speed is obtained from the relational expression, and the obtained brake distance is used as a predetermined lower limit of the three-dimensional distance. 3. The monitoring method according to claim 1, wherein a position measuring device by a satellite navigation system and a wireless transmitter are attached to the heavy equipment, and the receiving station is provided with a wireless receiver for receiving from the wireless transmitter. Moving the heavy equipment to a plurality of reference points in the field of view of the imager pair and measuring the three-dimensional coordinates in the construction area by the position measuring device at each reference point, transmitting the measurement signal from the wireless transmitter, The stereo image measurement method is converted by comparing the three-dimensional coordinates received by the radio receiver of the receiving station with the two-dimensional coordinates of the reference point image on the stereo image pair received by the image receiver of the receiving station. Departure monitoring method for heavy equipment for remote construction that defines parameters. 4. A method according to claim 1, wherein each optical axis of said pair of image pickup devices is orthogonal to each other. 5. The monitoring method according to claim 1, wherein an optotype of a predetermined color is attached to the heavy equipment, and the image of the predetermined color is extracted from the stereo image pair. A departure monitoring method for a heavy equipment for remote construction, comprising detecting two-dimensional coordinates of an image of the heavy equipment. 6. The monitoring method according to claim 5, wherein a plurality of said heavy machines are arranged in a construction area, and targets of different colors are attached to the respective heavy machines.