JP7135191B2 - Working machine measurement system, working machine, and working machine measurement method - Google Patents

Description

The present invention relates to a work machine measurement system, a work machine, and a work machine measurement method.

2. Description of the Related Art In the technical field related to working machines, a working machine equipped with an imaging device, such as that disclosed in Patent Document 1, is known. A pair of images captured by a pair of imaging devices are subjected to stereo processing to measure the three-dimensional shape of the construction target around the work machine. Patent Literature 1 discloses capturing images for stereo processing while a rotating body is rotating.

WO2017/033991

Japanese Patent Application Laid-Open No. 2002-200002 discloses the time point of starting the shooting, but does not disclose a method for determining the time point of stopping the shooting. Therefore, for example, if the time to stop imaging is determined based on a predetermined elapsed time from the start of imaging, the imaging may be stopped before the imaging of all areas that the operator is going to measure in stereo is completed. It may get lost. In such a case, the operator needs to perform stereo measurement again for the remaining area, which complicates the work.

An object of the present invention is to smoothly obtain a three-dimensional shape of a construction target.

According to an aspect of the present invention, an image acquisition unit that acquires an image of a construction target captured by an imaging device mounted on a revolving body of a work machine while the revolving body is revolving; a determining unit that determines whether or not the photographing stop condition is satisfied based on the turning data of the rotating body; and an output unit that outputs a photographing stop command signal based on the determination result of the determining unit. A work machine metrology system is provided.

ADVANTAGE OF THE INVENTION According to the aspect of this invention, the three-dimensional shape of construction object can be acquired smoothly.

FIG. 1 is a perspective view showing an example of a working machine according to this embodiment. FIG. 2 is a perspective view showing a part of the working machine according to this embodiment. FIG. 3 is a functional block diagram showing an example of the measurement system according to this embodiment. FIG. 4 is a diagram schematically showing an example of the operation of the work machine according to this embodiment. FIG. 5 is a diagram schematically showing an example of the operation of the measurement system according to this embodiment. FIG. 6 is a schematic diagram for explaining an example of processing of the measurement system according to this embodiment. FIG. 7 is a schematic diagram for explaining an example of processing of the measurement system according to this embodiment. FIG. 8 is a schematic diagram for explaining an example of processing of the measurement system according to this embodiment. FIG. 9 is a perspective view schematically showing an example of the input device according to this embodiment. FIG. 10 is a flow chart showing an example of the measurement method according to this embodiment. FIG. 11 is a timing chart of the measurement method according to this embodiment.

Embodiments of the present invention will be described below with reference to the drawings, but the present invention is not limited thereto. The constituent elements of the embodiments described below can be combined as appropriate. Also, some components may not be used.

In the following description, a three-dimensional field coordinate system (Xg, Yg, Zg), a three-dimensional vehicle body coordinate system (Xm, Ym, Zm), and a three-dimensional imaging device coordinate system (Xs, Ys, Zs) are The positional relationship of each part will be explained.

The field coordinate system is a coordinate system referenced to an earth-fixed origin. The field coordinate system is a coordinate system defined by GNSS (Global Navigation Satellite System). GNSS refers to the Global Navigation Satellite System. An example of the global navigation satellite system is GPS (Global Positioning System).

The field coordinate system is defined by a horizontal Xg-axis, a horizontal Yg-axis orthogonal to the Xg-axis, and a Zg-axis orthogonal to the Xg-axis and the Yg-axis. The rotation or tilting direction about the Xg axis is the θXg direction, the rotation or tilting direction about the Yg axis is the θYg direction, and the rotation or tilting direction about the Zg axis is the θZg direction. The Zg-axis direction is the vertical direction.

The vehicle body coordinate system includes the Xm axis on a first predetermined plane based on the origin defined on the vehicle body of the working machine, the Ym axis on the first predetermined plane perpendicular to the Xm axis, and the Zm axis perpendicular to the Xm and Ym axes. axis and The rotation or tilting direction about the Xm axis is the θXm direction, the rotation or tilting direction about the Ym axis is the θYm direction, and the rotation or tilting direction about the Zm axis is the θZm direction. The Xm-axis direction is the longitudinal direction of the work machine, the Ym-axis direction is the vehicle width direction of the work machine, and the Zm-axis direction is the vertical direction of the work machine.

The imaging device coordinate system includes the Xs axis of a second predetermined plane based on the origin defined in the imaging device, the Ys axis of the second predetermined plane orthogonal to the Xs axis, and the Zs axis orthogonal to the Xs axis and the Ys axis. and The rotation or tilting direction about the Xs axis is the θXs direction, the rotation or tilting direction about the Ys axis is the θYs direction, and the rotation or tilting direction about the Zs axis is the θZs direction. The Xs-axis direction is the vertical direction of the imaging device, the Ys-axis direction is the width direction of the imaging device, and the Zs-axis direction is the front-back direction of the imaging device. The Zs-axis direction is parallel to the optical axis of the optical system of the imaging device.

The position in the field coordinate system, the position in the vehicle body coordinate system, and the position in the imaging device coordinate system can be mutually converted.

First embodiment.
[Working machine]
FIG. 1 is a perspective view showing an example of a working machine 1 according to this embodiment. In this embodiment, an example in which the work machine 1 is a hydraulic excavator will be described. In the following description, the work machine 1 is appropriately called the hydraulic excavator 1 .

As shown in FIG. 1, the hydraulic excavator 1 has a vehicle body 1B and a work implement 2. As shown in FIG. The vehicle body 1B has a revolving body 3 and a running body 5 that supports the revolving body 3 so as to be able to turn.

The revolving body 3 has an operator's cab 4 . A hydraulic pump and an internal combustion engine are arranged on the revolving body 3 . The revolving body 3 can be revolved around a revolving axis Zr. The turning axis Zr is parallel to the Zm axis of the vehicle body coordinate system. The origin of the vehicle body coordinate system is defined, for example, at the center of the swing circle of the revolving body 3 . The center of the swing circle is located at the pivot axis Zr of the pivot 3 .

The traveling body 5 has crawler belts 5A and 5B. The hydraulic excavator 1 travels by rotating the crawler belts 5A and 5B. The Zm-axis of the vehicle body coordinate system is orthogonal to the contact surfaces of crawler belts 5A and 5B. The upward direction (+Zm direction) of the vehicle body coordinate system is the direction away from the contact surface of the crawler belts 5A and 5B, and the downward direction (−Zm direction) of the vehicle body coordinate system is the direction opposite to the upward direction of the vehicle body coordinate system.

The work machine 2 is connected to the revolving body 3 . At least part of the work implement 2 is arranged forward of the revolving body 3 in the vehicle body coordinate system. The front of the vehicle body coordinate system (+Xm direction) is the direction in which the work implement 2 exists with respect to the revolving body 3, and the rearward direction of the vehicle body coordinate system (-Xm direction) is the direction opposite to the front of the vehicle body coordinate system. be.

The work machine 2 drives a boom 6 connected to the revolving body 3, an arm 7 connected to the boom 6, a bucket 8 connected to the arm 7, a boom cylinder 10 that drives the boom 6, and the arm 7. and an arm cylinder 11 for driving the bucket 8, and a bucket cylinder 12 for driving the bucket 8. The boom cylinder 10, the arm cylinder 11, and the bucket cylinder 12 are hydraulic cylinders driven by hydraulic pressure.

The hydraulic excavator 1 also includes a position detection device 23 that detects the position of the revolving body 3 , an attitude detection device 24 that detects the attitude of the revolving body 3 , and a control device 40 .

The position detection device 23 detects the position of the revolving superstructure 3 in the field coordinate system. The position of the revolving body 3 includes Xg-axis direction coordinates, Yg-axis direction coordinates, and Zg-axis direction coordinates. Position detection device 23 includes a GPS receiver. The position detection device 23 is provided on the revolving body 3 .

A GPS antenna 21 is provided on the revolving body 3 . Two GPS antennas 21 are arranged, for example, in the Ym-axis direction of the vehicle body coordinate system. The GPS antenna 21 receives radio waves from GPS satellites and outputs a signal generated based on the received radio waves to the position detection device 23 . The position detection device 23 detects the position of the GPS antenna 21 in the field coordinate system based on the signal from the GPS antenna 21 .

The position detection device 23 performs arithmetic processing based on at least one of the positions of the two GPS antennas 21 to calculate the position of the revolving superstructure 3 . The position of the revolving body 3 may be the position of one GPS antenna 21 or a position between the position of one GPS antenna 21 and the position of the other GPS antenna 21 .

The posture detection device 24 detects the posture of the revolving superstructure 3 in the field coordinate system. The posture of the revolving structure 3 is defined by a roll angle indicating the inclination angle of the revolving structure 3 in the direction of rotation about the Xm axis, a pitch angle indicating the angle of inclination of the revolving structure 3 in the direction of rotation about the Ym axis, and Zm and an azimuth angle indicating the tilt angle of the revolving body 3 in the direction of rotation about the axis. The attitude detection device 24 includes an inertial measurement unit (IMU). The posture detection device 24 is provided on the revolving body 3 . A gyro sensor may be mounted on the revolving body 3 as the attitude detection device 24 .

The posture detection device 24 detects acceleration and angular velocity acting on the posture detection device 24 . By detecting the acceleration and angular velocity acting on the posture detection device 24, the acceleration and angular velocity acting on the revolving body 3 are detected. The attitude detection device 24 performs arithmetic processing based on the acceleration and angular velocity acting on the revolving body 3 to calculate the attitude of the revolving body 3 including the roll angle, pitch angle, and azimuth angle.

Note that the azimuth angle may be calculated based on the detection data of the position detection device 23 . The position detection device 23 can calculate the azimuth angle of the revolving superstructure 3 with respect to the reference azimuth in the site coordinate system based on the position of one GPS antenna 21 and the position of the other GPS antenna 21 . The reference direction is north, for example. The position detection device 23 calculates a straight line connecting the position of one GPS antenna 21 and the position of the other GPS antenna 21, and based on the angle formed by the calculated straight line and the reference direction, determines the position of the revolving structure 3 with respect to the reference direction. Azimuth angle can be calculated.

Next, the stereo camera 300 according to this embodiment will be described. FIG. 2 is a perspective view showing a portion of the hydraulic excavator 1 according to this embodiment. As shown in FIG. 2 , the hydraulic excavator 1 has a stereo camera 300 . The stereo camera 300 is a camera capable of measuring the distance to the construction target SB by simultaneously capturing images of the construction target SB from a plurality of directions and generating parallax data.

The stereo camera 300 photographs the construction target SB around the hydraulic excavator 1 . The construction target includes an excavation construction target to be excavated by the working machine 2 of the hydraulic excavator 1 . The construction target SB may be a construction target to be constructed by a working machine other than the hydraulic excavator 1, or may be a construction target to be constructed by a worker. The construction target SB is a concept including a construction target before construction, a construction target during construction, and a construction target after construction.

A stereo camera 300 is mounted on the revolving body 3 . Stereo camera 300 is provided in driver's cab 4 . The stereo camera 300 is arranged in front of the driver's cab 4 (+Xm direction) and above (+Zm direction). The stereo camera 300 photographs the photographing target SB in front of the hydraulic excavator 1 .

Stereo camera 300 has a plurality of imaging devices 30 . The imaging device 30 is mounted on the revolving body 3 . The imaging device 30 has an optical system and an image sensor. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In this embodiment, the imaging device 30 includes four imaging devices 30A, 30B, 30C, and 30D.

A pair of imaging devices 30 constitute a stereo camera 300 . The stereo camera 300 includes a first stereo camera 301 composed of a pair of imaging devices 30A and 30B and a second stereo camera 302 composed of a pair of imaging devices 30C and 30D.

The imaging devices 30A and 30C are arranged on the +Ym side (working machine 2 side) of the imaging devices 30B and 30D. The imaging device 30A and the imaging device 30B are arranged with an interval in the Ym-axis direction. The imaging device 30C and the imaging device 30D are arranged with a gap in the Ym-axis direction. The imaging devices 30A and 30B are arranged on the +Zm side of the imaging devices 30C and 30D. In the Zm-axis direction, the imaging devices 30A and 30B are arranged at substantially the same position. In the Zm-axis direction, the imaging device 30C and the imaging device 30D are arranged at substantially the same position.

The imaging devices 30A and 30B face upward (+Zm direction). The imaging devices 30C and 30D face downward (-Zm direction). Also, the imaging devices 30A and 30C face forward (+Xm direction). The imaging devices 30B and 30D face slightly toward the +Ym side (the work implement 2 side) rather than toward the front. That is, the imaging devices 30A and 30C face the front of the revolving body 3, and the imaging devices 30B and 30D face the imaging devices 30A and 30C. The imaging devices 30B and 30D may face the front of the revolving body 3, and the imaging devices 30A and 30C may face the imaging devices 30B and 30D.

The image capturing device 30 captures an image of the construction target SB present in front of the revolving body 3 . A pair of images captured by a pair of imaging devices 30 are subjected to stereo processing in the control device 40 to calculate three-dimensional data representing the three-dimensional shape of the construction target SB. The control device 40 converts the three-dimensional data of the construction target SB in the imaging device coordinate system into three-dimensional data of the construction target SB in the site coordinate system. The three-dimensional data indicates the three-dimensional position of the construction target SB. The three-dimensional position of the construction target SB includes three-dimensional coordinates of each of a plurality of parts on the surface of the construction target SB.

An imaging device coordinate system is defined for each of the plurality of imaging devices 30 . The imaging device coordinate system is a coordinate system based on the origin fixed to the imaging device 30 . The Zs axis of the imaging device coordinate system coincides with the optical axis of the optical system of the imaging device 30 .

In this embodiment, two sets of stereo cameras (first stereo camera 301 and second stereo camera 302) are mounted on revolving body 3, but one set of stereo cameras may be mounted. However, three or more sets of stereo cameras may be mounted.

Moreover, as shown in FIG. 2, the hydraulic excavator 1 has a driver's seat 4S, an input device 32, and an operation device 35. As shown in FIG. A driver's seat 4</b>S, an input device 32 , and an operating device 35 are arranged in the driver's cab 4 . A driver of the hydraulic excavator 1 sits on the driver's seat 4S.

The input device 32 is operated by the driver to start or stop imaging by the imaging device 30 . The input device 32 is provided near the driver's seat 4S. The imaging by the imaging device 30 is started or stopped by operating the input device 32 . The start of photographing means that photographing is started when the imaging device 30 acquires a photographing start command signal, which will be described later, and the photographed image is transferred to the storage device 42 from the time when the signal acquisition unit 411 acquires the photographing start command signal. and executing stereo processing based on the images captured from the time when the signal acquisition unit 411 acquired the imaging start command signal.

Similarly, to stop photographing means to stop photographing when the imaging device 30 acquires the photographing stop command signal, and to transfer the photographed image to the storage device 42 from the time when the signal acquisition unit 411 acquires the photographing stop command signal. and stopping the execution of the stereo processing based on the captured image from the time when the signal acquisition unit 411 acquires the imaging stop command signal.

The operation device 35 is operated by the driver to drive or stop the work machine 2, to rotate or stop the rotation of the rotating body 3, that is, to perform a rotating operation, and to travel or stop traveling, that is, to perform a traveling operation of the traveling body 5. The operating device 35 includes a right operating lever 35R and a left operating lever 35L for operating the work implement 2 and the revolving body 3. As shown in FIG. The operation device 35 also includes a right travel lever and a left travel lever (not shown) for operating the traveling body 5 . By operating the operating device 35, the work machine 2 is driven or stopped, the revolving body 3 is turned, and the traveling body 5 is driven.

[Measurement system]
Next, the measurement system 50 according to this embodiment will be described. FIG. 3 is a functional block diagram showing an example of the measurement system 50 according to this embodiment. A measurement system 50 is provided in the hydraulic excavator 1 .

The measurement system 50 includes a control device 40, a stereo camera 300 including a first stereo camera 301 and a second stereo camera 302, a position detection device 23, an orientation detection device 24, and an operation for detecting operation data of an operation device 35. A volume sensor 36 and an input device 32 are provided.

The control device 40 is provided on the revolving body 3 of the hydraulic excavator 1 . Controller 40 includes a computer system. The control device 40 includes an arithmetic processing device 41 including a processor such as a CPU (Central Processing Unit), and a volatile memory such as a RAM (Random Access Memory) and a non-volatile memory such as a ROM (Read Only Memory). It has a storage device 42 and an input/output interface 43 .

The arithmetic processing unit 41 includes an image acquisition unit 410, a signal acquisition unit 411, a common part extraction unit 413, a turning data acquisition unit 415, an imaging position calculation unit 416, a three-dimensional position calculation unit 417, and a determination unit 418. and

The image acquisition unit 410 acquires a plurality of images PC of the construction target SB captured by the imaging device 30 while the hydraulic excavator 1 is turning. The image acquisition unit 410 also acquires an image PC of the construction target SB captured by the imaging device 30 when the hydraulic excavator 1 is in a non-operating state, that is, in a state in which neither traveling nor turning is stopped.

The signal acquisition unit 411 acquires a command signal generated by operating the input device 32 . The input device 32 is operated to start or stop imaging by the imaging device 30 . The command signal includes a command signal to start photographing and a command signal to stop photographing. The signal acquisition unit 411 outputs the acquired imaging start command signal and imaging stop command signal to the imaging device 30 .

Note that the image PC captured during the period between when the signal acquisition unit 411 acquires the imaging start command signal and when the imaging stop command signal is acquired is stored in the storage device 42, and is stored in the storage device 42. An image PC may be used for stereo processing.

The common part extraction unit 413 extracts a common part KS of a plurality of images PC captured by the imaging device 30 while the hydraulic excavator 1 is turning. The common portion KS will be described later.

The turning data acquisition unit 415 acquires turning data of the turning body 3 during turning. The turning data is data indicating the turning state of the turning body 3 and includes at least one of the turning speed V, turning angle θ, and turning direction RD of the turning body 3 . As will be described later, in this embodiment, the turning data is calculated by the imaging position calculator 416 . The turning data acquisition unit 415 acquires turning data from the imaging position calculation unit 416 .

The imaging position calculation unit 416 calculates the position P and orientation of the imaging device 30 at the time of shooting. When the revolving body 3 is revolving, the position P of the imaging device 30 during photographing includes the position of the imaging device 30 in the revolving direction RD. When the running body 5 is in a running state, the position P of the imaging device 30 at the time of photographing includes the position of the imaging device 30 in the running direction MD. Also, the imaging position calculation unit 416 calculates turning data of the turning body 3 . The turning data of the turning body 3 includes at least one of the turning velocity V, turning angle θ, and turning direction RD of the turning body 3 .

The three-dimensional position calculation unit 417 performs stereo processing on the pair of images PC captured by the pair of imaging devices 30 to calculate the three-dimensional position of the construction target SB in the imaging device coordinate system. The three-dimensional position calculation unit 417 calculates the three-dimensional position of the construction target SB in the imaging device coordinate system based on the position P of the imaging device 30 calculated by the imaging position calculation unit 416. Convert to position.

The determination unit 418 determines whether or not a condition for stopping photographing of the image PC used for stereo processing is satisfied based on the turning data. If the stop condition is not met, the capture of the image PC for stereo processing continues. The stop conditions are stored in the stop condition storage unit 422 of the storage device 42 . A stop condition will be described later.

The output unit 419 outputs a shooting stop command signal based on the determination result of the determination unit 418 . The output unit 419 outputs a command signal to stop shooting to the signal acquisition unit 411 or the imaging device 30 when the determination unit 418 determines that the condition for stopping shooting of the image PC is satisfied. The output unit 419 may output the imaging stop command signal to other devices. The determination result includes a result that the condition for stopping photographing of the image PC is satisfied and a result that the condition for stopping photographing of the image PC is not satisfied.

The storage device 42 has a stop condition storage section 422 and an image storage section 423 .

The input/output interface 43 includes an interface circuit that connects the arithmetic processing device 41 and the storage device 42 to external devices. The input/output interface 43 is connected with the hub 31 , the position detection device 23 , the attitude detection device 24 , the operation amount sensor 36 and the input device 32 .

A plurality of imaging devices 30 ( 30 A, 30 B, 30 C, 30 D) are connected to an arithmetic processing device 41 via a hub 31 . The imaging device 30 takes an image PC of the construction target SB based on the imaging start command signal from the signal acquisition unit 411 . An image PC of the construction target SB photographed by the imaging device 30 is input to the arithmetic processing device 41 and the storage device 42 via the hub 31 and the input/output interface 43, respectively. Each of the image acquisition unit 410 and the image storage unit 423 acquires the image PC of the construction target SB captured by the imaging device 30 via the hub 31 and the input/output interface 43 . Note that the hub 31 may be omitted.

The operation amount sensor 36 detects operation data of the operation device 35 as described above. The operation amount sensor 36 may detect an operation state in which the operation device 35 is operated and a neutral state in which the operation device 35 is not operated based on the detected operation data. The operation data includes the operation amount of the operation device 35 and the detection results of the operation state and the neutral state. The operation amount sensor 36 may detect the operation data based on the operation position of the operation lever, or detect the operation data from the pilot oil pressure generated by operating the operation lever, that is, PPC (Pressure Proportional Control) pressure. Alternatively, the operation data may be detected by other methods.

The input device 32 is operated to start or stop imaging by the imaging device 30 . By operating the input device 32, an imaging start command signal or an imaging stop command signal is generated. At least one of an operation switch, an operation button, a touch panel, a voice input, and a keyboard is exemplified as the input device 32 . The operation of the input device 32 includes voice input.

[Turning terrain measurement]
Next, an example of the operation of the hydraulic excavator 1 according to this embodiment will be described. FIG. 4 is a diagram schematically showing an example of the operation of the hydraulic excavator 1 according to this embodiment. The measurement system 50 continuously captures images PC of the construction target SB around the hydraulic excavator 1 with the imaging device 30 while the revolving body 3 is revolving. The imaging device 30 sequentially captures images PC of the construction target SB at a predetermined cycle while the revolving body 3 is revolving.

The imaging device 30 is mounted on the revolving body 3 . As the revolving body 3 revolves, the photographing area FM of the imaging device 30 moves in the revolving direction RD. The imaging device 30 continuously captures the images PC of the construction target SB while the revolving body 3 is turning, whereby the imaging device 30 can acquire the respective images PC of a plurality of areas of the construction target SB. The three-dimensional position calculation unit 417 can calculate the three-dimensional position of the construction target SB around the hydraulic excavator 1 by stereo-processing the pair of images PC captured by the pair of imaging devices 30 .

The three-dimensional position of the construction target SB calculated by the stereo processing is defined in the imaging device coordinate system. A three-dimensional position calculation unit 417 transforms the three-dimensional position in the imaging device coordinate system into a three-dimensional position in the field coordinate system. In order to transform the three-dimensional position in the imaging device coordinate system into the three-dimensional position in the field coordinate system, the position and orientation of the rotating body 3 in the field coordinate system are required. The position and orientation of the revolving superstructure 3 in the site coordinate system can be detected by the position detection device 23 and the orientation detection device 24 .

While the hydraulic excavator 1 is swinging, the position detection device 23 and the attitude detection device 24 mounted on the hydraulic excavator 1 are displaced. Detection data output from each of the position detection device 23 and the attitude detection device 24 in a moving state may be unstable or the detection accuracy may be lowered.

The position detection device 23 outputs detection data at a predetermined cycle. Therefore, when the detection of the position by the position detection device 23 and the imaging by the imaging device 30 are performed in parallel while the hydraulic excavator 1 is turning, the imaging device 30 captures an image when the position detection device 23 is moving. There is a possibility that the timing of detecting the position and the timing of detecting the position by the position detecting device 23 are not synchronized. If the three-dimensional position of the construction target SB is coordinate-transformed based on the detection data of the position detection device 23 detected at a timing different from the timing of photographing, the measurement accuracy of the three-dimensional position may decrease.

In the present embodiment, the measurement system 50 highly accurately calculates the position and orientation of the revolving superstructure 3 at the time when the imaging device 30 captures the image PC while the revolving superstructure 3 is revolving, based on a method to be described later. Thereby, the measurement system 50 can highly accurately calculate the three-dimensional position of the construction target SB in the site coordinate system.

The imaging position calculation unit 416 acquires detection data of each of the position detection device 23 and the posture detection device 24 detected when the hydraulic excavator 1 is in the operation stop state. There is a high possibility that the detection data detected by the position detection device 23 and the attitude detection device 24 are stable when the hydraulic excavator 1 is in a non-operating state. The imaging position calculation unit 416 acquires the detection data detected in each of the operation stop states before the turning body 3 starts turning and after turning stops. The imaging device 30 captures an image PC of the construction target SB in each operation stop state before the revolving body 3 starts revolving and after revolving stops. The imaging position calculation unit 416 calculates an image PC captured while the hydraulic excavator 1 is in the non-operation state based on the detection data of the position detection device 23 and the posture detection device 24 acquired when the hydraulic excavator 1 is in the non-operation state. The calculated three-dimensional position of the construction target SB in the imaging device coordinate system is transformed into a three-dimensional position in the site coordinate system.

On the other hand, when the image PC of the construction target SB is captured by the imaging device 30 while the revolving body 3 is revolving, the imaging position calculation unit 416 calculates the image PC of the construction target SB by the imaging device 30 while the revolving body 3 is revolving based on a method described later. The position and attitude of the revolving body 3 at the time when the image PC captures the image is calculated. The imaging position calculation unit 416 converts the three-dimensional position in the imaging device coordinate system calculated from the photographed image PC into a three-dimensional position in the field coordinate system based on the calculated position and orientation of the revolving body 3 .

FIG. 5 is a diagram schematically showing an example of the operation of the measurement system 50 according to this embodiment. FIG. 5 is a schematic diagram for explaining that the imaging device 30 photographs the construction target SB while the revolving body 3 is revolving.

In the following description, it is assumed that the traveling body 5 is in a stopped state. As shown in FIG. 5, when the revolving body 3 revolves, the imaging device 30 mounted on the revolving body 3 and the imaging area FM of the imaging device 30 move in the revolving direction RD. The imaging area FM of the imaging device 30 is defined based on the field of view of the optical system of the imaging device 30 . The imaging device 30 acquires an image PC of the construction target SB arranged in the imaging area FM. As the revolving body 3 turns, the photographing area FM of the imaging device 30 moves in the turning direction RD of the revolving body 3 . The imaging device 30 captures images PC of the construction target SB sequentially arranged in the moving imaging region FM.

FIG. 5 shows an example in which the photographing area FM moves in the turning direction RD in order of the photographing area FM1, the photographing area FM2, and the photographing area FM3 as the revolving body 3 turns. The imaging area FM1 is defined at a first position PJ1 in the turning direction RD. The imaging area FM2 is defined at a second position PJ2 in the turning direction RD. The imaging area FM3 is defined at a third position PJ3 in the turning direction RD. The second position PJ2 is a position turned by a turning angle Δθ1 from the first position PJ1. The third position PJ3 is a position turned by a turning angle Δθ2 from the second position PJ2. The imaging device 30 captures an image PC1 of the construction target SB placed in the imaging area FM1, an image PC2 of the construction target SB placed in the imaging area FM2, and an image PC3 of the construction target SB placed in the imaging area FM3. to shoot. The image PC1, the image PC2, and the image PC3 are images captured by the same imaging device 30 (the imaging device 30C in the example shown in FIG. 5).

The image capturing device 30 captures images at a predetermined timing so that the overlapping areas OB are provided in the adjacent image capturing areas FM. FIG. 5 shows an example in which an overlapping area OB1 is provided between the imaging areas FM1 and FM2, and an overlapping area OB2 is provided between the imaging areas FM2 and FM3. The overlapping area OB1 is an overlapping area OB where the image PC1 and a part of the image PC2 overlap. The overlapping area OB2 is an overlapping area OB where the image PC2 and a part of the image PC3 overlap.

A common portion KS of the images PC is present in the overlapping area OB. A common portion KS1 existing in the overlap region OB1 is a common portion KS between the image PC1 and the image PC2. A common portion KS2 existing in the overlap region OB2 is a common portion KS between the images PC2 and PC3. The common part extraction unit 413 extracts a common part KS of a plurality of two-dimensional images PC captured by the imaging device 30 .

FIG. 6 is a schematic diagram for explaining an example of processing of the measurement system 50 according to this embodiment. FIG. 6 is a diagram showing an example of images PC (PC1, PC2) captured while the revolving body 3 is revolving. A common part extraction unit 413 extracts a common part KS from the image PC.

As shown in FIG. 6, the common part extraction unit 413 extracts the image PC1 and the image PC2 from the image PC1 of the construction target SB arranged in the imaging region FM1 and the image PC2 of the construction target SB arranged in the imaging region FM2. and extract the common part KS1. The imaging position calculator 416 calculates the estimated angle θs (Δθ1 in FIG. 5) of the revolving body 3 based on the common portion KS1 extracted by the common portion extractor 413 . Also, the imaging position calculation unit 416 calculates an estimated position Ps (PJ2 in FIG. 5) of the imaging device 30 at the time of shooting based on the estimated angle θs.

The common portion KS is a feature point in the image PC. A common portion extractor 413 extracts a common portion KS based on a known feature point detection algorithm such as ORB (Oriented FAST and Rotated BRIEF) or Harris corner detection. The common portion extraction unit 413 extracts a plurality of feature points from each of the plurality of images PC, and searches for similar feature points from the extracted plurality of feature points, thereby extracting a common portion KS. The common part extraction unit 413 may extract a plurality of common parts KS. The common part extraction unit 413 extracts, for example, corners of the construction target SB in the image PC as feature points.

The rotation of the rotating body 3 moves the photographing area FM of the imaging device 30 in the turning direction RD, and the movement of the photographing area FM displaces the common portion KS in the image PC. In the example shown in FIG. 6, the common portion KS exists at the pixel position PX1 in the image PC1 and exists at the pixel position PX2 in the image PC2. The position where the common portion KS exists is different between the image PC1 and the image PC2. That is, the common portion KS1 is displaced between the image PC1 and the image PC2. The imaging position calculator 416 can calculate the turning angle Δθ1 from the position PJ1 to the position PJ2 based on the position of the common portion KS in the multiple images PC.

FIG. 7 is a schematic diagram for explaining an example of processing of the measurement system 50 according to this embodiment. FIG. 7 is a diagram for explaining how the imaging apparatus coordinate system (Xs, Ys, Zs) moves as the revolving body 3 revolves. As shown in FIG. 7, the revolving body 3 revolves around the revolving axis Zr within the Xm-Ym plane of the vehicle body coordinate system. When the revolving body 3 turns by a turning angle Δθ around the turning axis Zr, the imaging apparatus coordinate system (Xs, Ys, Zs) moves by a turning angle Δθ around the turning axis Zr. The imaging position calculation unit 416 calculates the estimated angle θs of the revolving structure 3 and the angle of the imaging device 30 when the revolving structure 3 turns by the revolving angle Δθ from the pre-turning angle θra under the constraint condition that the revolving structure 3 revolves around the revolving axis Zr. An estimated position Ps can be calculated.

[Calculation of estimated angle]
FIG. 8 is a schematic diagram for explaining an example of processing of the measurement system 50 according to this embodiment. FIG. 8 is a schematic diagram showing an example of calculating the turning angle θ of the turning body 3 and the position P of the imaging device 30. As shown in FIG. FIG. 8 shows the turning angle θ of the turning body 3 and the position P of the imaging device 30 in the vehicle body coordinate system.

In the example shown in FIG. 8, the revolving structure 3 turns in the turning direction RD in order of the pre-turning angle θra, the first estimated angle θs1, the second estimated angle θs2, the third estimated angle θs3, and the post-turning angle θrb. As the revolving body 3 turns, the imaging device 30 moves to a pre-turning position Pra, a first estimated position Ps1, a second estimated position Ps2, a third estimated position Ps3, and a post-turning position Prb in this order. The imaging region FM of the imaging device 30 moves in order of imaging region FMra, imaging region FMs1, imaging region FMs2, imaging region FMs3, and imaging region FMrb as the imaging device 30 moves in the turning direction RD.

At the pre-swing angle θra and the post-swing angle θrb, the revolving body 3 is in a revolving stop state. The revolving structure 3 in the state of stopping revolving at the pre-swing angle θra passes through the first estimated angle θs1, the second estimated angle θs2, and the third estimated angle θs3, and reaches the post-swing angle θrb. It turns from θra to the post-turning angle θrb.

The image pickup device 30 acquires an image PCra of the construction target SB arranged in the photographing area FMra while being arranged at the pre-turn position Pra. The imaging device 30 acquires an image PCs1 of the construction target SB placed in the imaging region FMs1 while being placed at the first estimated position Ps1. The imaging device 30 acquires an image PCs2 of the construction target SB placed in the imaging region FMs2 while being placed at the second estimated position Ps2. The image pickup device 30 acquires an image PCs3 of the construction target SB arranged in the photographing area FMs3 while being arranged at the third estimated position Ps3. The image pickup device 30 acquires an image PCrb of the construction target SB arranged in the photographing region FMrb while being arranged at the post-turn position Prb.

As described above, the imaging position calculation unit 416 calculates the position and orientation of the revolving superstructure 3 detected by the position detection device 23 and the orientation detection device 24 when the revolving superstructure 3 is in a stop state before revolving. Calculate the angle θra. Further, the imaging position calculation unit 416 calculates the post-rotation angle θrb based on the position and attitude of the revolving body 3 detected by the position detection device 23 and the attitude detection device 24 in the revolving stopped state after the revolving body 3 stops revolving. calculate. Further, the imaging position calculation unit 416 calculates the pre-turning position Pra based on the pre-turning angle θra. Further, the imaging position calculation unit 416 calculates the post-turning position Prb based on the post-turning angle θrb.

The pre-turning angle θra, the post-turning angle θrb, the pre-turning position Pra, and the post-turning position Prb are calculated with high accuracy based on the detection data of the position detection device 23 and the detection data of the attitude detection device 24 .

As described above, the imaging position calculation unit 416 calculates the estimated angle θs (first estimated angle θs1, second estimated angle θs2, and a third estimated angle θs3). Also, the imaging position calculation unit 416 calculates the estimated position Ps (the first estimated position Ps1, the second estimated position Ps2, and the third estimated position Ps3) based on the estimated angle θs.

A common portion extraction unit 413 extracts a common portion KS1 between the image PCra and the image PCs1. The imaging position calculation unit 416, under the constraint condition that the revolving body 3 revolves around the revolving axis Zr, calculates the revolving motion based on the angle θra of the revolving body 3 before turning and the common portion KS1 between the image PCra and the image PCs1. Calculate the angle Δθ1. The first estimated angle θs1 is the sum of the correct angle θra before turning and the turning angle Δθ1 (θs1=θra+Δθ1).

Similarly, the common part extraction unit 413 can calculate the turning angle Δθ2, the turning angle Δθ3, and the turning angle Δθ4, and the imaging position calculation unit 416 can calculate the second estimated angle θs2 (θs2=θra+Δθ1+Δθ2), the third estimated angle An angle θs3 (θs3=θra+Δθ1+Δθ2+Δθ3) and a fourth estimated angle θs4 (θs4=θra+Δθ1+Δθ2+Δθ3+Δθ4) can be calculated.

In this way, the imaging position calculation unit 416 can stepwise calculate the estimated angles θs (θs1, θs2, θs3, θs4) of the revolving structure 3 when the imaging device 30 captures images while the revolving structure 3 is revolving. . Further, by calculating the estimated angle θs of the revolving structure 3, the imaging position calculation unit 416 calculates the estimated position Ps (Ps1, Ps2, Ps3 , Ps4) can be calculated.

The imaging position calculation unit 416 also acquires from the imaging device 30 time point data indicating the time points at which each of the plurality of images PC (PCra, PCs1, PCs2, PCs3, PCs4, and PCrb) was captured. The imaging device 416 can calculate the turning speed V based on the times when each of the plurality of images PC is captured and the turning angles Δθ (Δθ1, Δθ2, Δθ3, Δθ4). For example, based on the time from when the image PCs1 was captured to when the image PCs2 was captured and the amount of movement from the estimated angle θs1 to the estimated angle θs2 when the image PCs1 was captured, It is possible to calculate the turning angle V when turning from the estimated angle θs1 to the estimated angle θs2. Further, the imaging device 416 can calculate the turning direction R based on the turning angle Δθ and the time when each of the plurality of images PC is captured.

As described above, in this embodiment, the imaging position calculation unit 416 calculates the turning data of the turning body 3 including the turning angle θ, turning speed V, and turning direction R based on the common portion KS of the plurality of images PC. can be calculated.

[Correction of estimated angle]
The post-turn angle θrb is calculated with high accuracy based on the detection data of the position detection device 23 and the detection data of the attitude detection device 24 . On the other hand, the estimated angle θs4 is calculated based on the amount of displacement of the common portion KS. If the estimated angle θs4 and the estimated position Ps4 calculated using the common portion KS are accurate, the difference between the estimated angle θs4 and the post-turning angle θrb is small, and the difference between the estimated position Ps4 and the post-turning position Prb is small.

On the other hand, errors in the estimated angle θs4 and the estimated position Ps4 calculated using the common portion KS may increase due to, for example, accumulated errors in the amount of displacement of the common portion KS. When the error between the estimated angle θs4 and the estimated position Ps4 is large, the difference between the estimated angle θs4 and the post-turning angle θrb becomes large, and the difference between the estimated position Ps4 and the post-turning position Prb becomes large.

When there is a difference between the estimated angle θs4 and the post-turning angle θrb, the imaging position calculation unit 416 calculates the angle θr (the pre-turning angle θra and the post-turning angle θrb) that is the turning angle θ in the state where the turning is stopped. The estimated angle θs (the first estimated angle θs1, the second estimated angle θs2, the third estimated angle θs3, and the fourth estimated angle θs4), which is the turning angle θ in , is corrected. Further, the imaging position calculation unit 416 corrects the estimated position Ps based on the corrected estimated angle θs.

The imaging position calculation unit 416 can accurately calculate the estimated position Ps1 of the imaging device 30 when the image PCs1 is captured based on the corrected estimated angle θs1′. Similarly, the imaging position calculation unit 416 can accurately calculate the estimated position Ps2 of the imaging device 30 when the image PCs2 is captured based on the corrected estimated angle θs2′, and the corrected estimated angle θs3′. , the estimated position Ps3 of the imaging device 30 when the image PCs3 was captured can be calculated with high accuracy. Therefore, the three-dimensional position calculation unit 417 calculates the construction target SB in the field coordinate system based on the highly accurately calculated estimated position Ps of the imaging device 30 at the time of shooting and the image PCs captured by the imaging device 30. A three-dimensional position can be calculated with high accuracy.

The three-dimensional position calculator 417 stereo-processes the pair of images PC captured by the pair of imaging devices 30 to calculate the three-dimensional position of the construction target SB.

The three-dimensional position calculation unit 417 stereo-processes a pair of images PCra photographed by a pair of imaging devices 30 arranged at the pre-rotation position Pra while the revolving body 3 is in a state where the revolving body 3 stops revolving before it starts revolving. The three-dimensional position of the construction target SB arranged in FMra is calculated.

Further, the three-dimensional position calculation unit 417 stereo-processes the pair of images PCs captured by the pair of imaging devices 30 arranged at the estimated position Ps while the revolving body 3 is turning, and obtains images PCs arranged in the photographing area FMs. Then, the three-dimensional position of the construction target SB is calculated. The three-dimensional position calculation unit 417 stereo-processes the pair of images PCs photographed by the pair of imaging devices 30 arranged at the estimated position Ps while the traveling body 5 is in a stopped state and the revolving body 3 is turning. , the three-dimensional position of the construction target SB placed in the photographing area FMs is calculated.

Further, the three-dimensional position calculation unit 417 stereo-processes a pair of images PCrb photographed by a pair of imaging devices 30 arranged at the post-rotation position Prb in the state where the rotating body 3 stops rotating after the rotation is stopped. A three-dimensional position of the construction target SB placed in the imaging region FMrb is calculated.

Further, the three-dimensional position calculation unit 417 calculates the three-dimensional position of the work target SB based on a plurality of images PC captured while the revolving body 3 is turning and when the revolving is stopped. Integrate the 3D positions of the SBs.

[Shooting stop conditions]
Next, conditions for stopping photographing will be described. The photographing stop condition refers to a condition for stopping the photographing of the image PC used for stereo processing. The stop condition is determined in advance and stored in the stop condition storage unit 422 of the storage device 42 . Based on the turning data acquired by the turning data acquiring unit 415, the determining unit 418 determines whether or not the shooting stop condition is satisfied. Further, the determination unit 418 determines whether or not the shooting stop condition is satisfied based on the operation data of the operation device 35 detected by the operation amount sensor 36 . When it is determined that the stop condition is satisfied, the imaging of the image PC for stereo processing is stopped.

In this embodiment, the stop condition storage unit 422 stores a stop condition indicating that the revolving structure 3 has stopped revolving as the first stop condition. The determination unit 418 determines whether or not the revolving superstructure 3 has stopped revolving based on the operation data or the revolving data. When the revolving body 3 stops revolving, the imaging of the image PC for stereo processing stops.

For example, when the operator of the hydraulic excavator 1 puts the operation device 35 in the neutral state to stop the swing, the swing of the swing body 3 stops, and at the time when the swing body 3 stops, the photographing of the image PC stops. . When the determination unit 418 determines that the revolving structure 3 has stopped revolving, the three-dimensional position calculation unit 417 performs stereo processing based on the image PC captured before the revolving structure 3 stops revolving. to calculate the three-dimensional position of the construction target SB.

When the driver places the operating device 35 in the neutral state to stop the swinging of the swinging body 3, it means that the driver intends to stop taking the image PC for stereo processing. Therefore, by stopping the photographing of the image PC when the revolving body 3 stops rotating, the driver can easily stop the photographing simply by setting the operating device 35 to the neutral state.

The determination unit 418 determines whether or not the operation data of the operating device 35 or the swing data of the swing structure 3 satisfies the first stop condition. When it is determined that the revolving structure 3 stops revolving, it is determined that the operation data or the revolving data satisfies the first stop condition. On the other hand, when it is determined that the revolving structure 3 continues to revolve, it is determined that the operation data or revolving data do not satisfy the first stop condition.

The stop condition storage unit 422 also stores a second stop condition, which indicates that a predetermined time has elapsed since the swinging body 3 stopped swinging. The determination unit 418 determines whether or not a predetermined period of time has elapsed since the revolving superstructure 3 stopped revolving, based on the operation data or the revolving data. When a predetermined period of time has elapsed since the revolving body 3 stopped revolving, the photographing of the image PC for stereo processing stops.

For example, after the operator of the hydraulic excavator 1 puts the operation device 35 in the neutral state and the revolving body 3 stops revolving, the photographing of the image PC stops after a predetermined time elapses from the revolving stop point. When the determination unit 418 determines that the rotation stop time after the rotation of the rotating body 3 has stopped has exceeded the predetermined time, the three-dimensional position calculation unit 417 calculates the image PC captured before the predetermined time has elapsed. Stereo processing is performed based on to calculate the three-dimensional position of the construction target SB.

When the driver puts the operating device 35 in the neutral state and stops the swinging of the swinging body 3, the swiveling amount of the swinging body 3 may not be sufficient and there may be a region to be photographed in the construction target SB. In addition, even if the rotation of the rotating body 3 is unintentionally stopped during the rotation of the rotating body 3 for some reason, an area to be photographed remains in the construction target SB. In such a case, if the photographing stops immediately when the swinging body 3 stops rotating, the rest of the region must be photographed again, which is troublesome. After the operator sets the operation device 35 to the neutral state and stops the swinging of the revolving body 3, the photographing is not stopped immediately, but after a predetermined period of time has passed. Even if there remains an area to be photographed, the driver can continue photographing by restarting the swinging of the swing body 3 before the predetermined time elapses. As a result, the trouble of having to perform imaging from the beginning again can be saved.

In addition, when the driver turns the revolving structure 3 to photograph a certain area and then puts the operation device 35 in the neutral state to stop the revolving of the revolving structure 3, the driver may not be able to photograph the photographed area for some reason. is determined, after turning is stopped, turning in the opposite direction is restarted before a predetermined period of time elapses, and photographing is continued, so that the region can be photographed again.

The determination unit 418 determines whether or not the operation data of the operating device 35 or the swing data of the swing structure 3 satisfies the second stop condition. When it is determined that the predetermined time has passed since the revolving structure 3 stopped revolving, it is determined that the operation data or the revolving data satisfies the second stop condition. On the other hand, when it is determined that the predetermined time has not elapsed since the revolving structure 3 stopped revolving, it is determined that the operation data or the revolving data does not satisfy the second stop condition.

The photographing is continued until a predetermined time elapses from the time when the revolving body 3 stops revolving. Further, even when the rotation stop time from the rotation stop of the rotation body 3 to the rotation restart is less than the predetermined time, the photographing is continued. The turning direction after turning is restarted may be the same as the turning direction (forward direction) before turning is stopped, or may be the opposite direction. When the determination unit 418 determines that the rotation stop time is less than the predetermined time, the three-dimensional position calculation unit 417 performs stereo processing based on the images PC captured before the rotation is stopped and after the rotation is restarted. to calculate the three-dimensional position of the construction target SB.

In addition, the stop condition storage unit 422 stores, as a third stop condition, a stop condition indicating that the signal acquisition unit 411 has acquired a shooting stop command signal generated by operating the input device 32 . ing. The determination unit 418 determines whether or not the command signal acquired by the signal acquisition unit 411 is a command signal to stop shooting. Shooting of the image PC for stereo processing is stopped at the time when the shooting stop command signal is acquired.

For example, when the operator of the hydraulic excavator 1 operates the input device 32 to generate a command signal to stop photographing, the photographing of the image PC is stopped. When the determination unit 418 determines that the signal acquisition unit 411 has acquired the image capturing stop command signal, the three-dimensional position calculation unit 417 performs a stereo image based on the image PC captured before the acquisition of the image capturing stop command signal. Processing is performed to calculate the three-dimensional position of the construction target SB.

The driver operates the input device 32 with the intention of stopping the imaging of the image PC for stereo processing. Therefore, by stopping the photographing of the image PC when the signal acquisition unit 411 receives the photographing stop command signal generated by operating the input device 32, the driver can operate the input device 32 only. , you can easily stop shooting at any time. For example, when the photographing stop condition is the first stop condition or the second stop condition, and the driver wants to photograph only a part of the area in the middle of turning while rotating the revolving structure 3, the driver must stop turning when the photographing of the part of the area stops. If the photographing stop condition is the third stop condition, the driver can photograph the partial area without stopping the turning operation in the middle.

The determination unit 413 determines whether or not the command signal acquired by the signal acquisition unit 411 satisfies the third stop condition. When it is determined that the command signal is a photographing stop command signal, it is determined that the third stop condition is satisfied. On the other hand, when it is determined that the command signal is not the photographing stop command signal, it is determined that the third stop condition is not satisfied.

In this embodiment, the determination unit 418 determines whether or not the imaging start condition is satisfied based on the command signal acquired by the signal acquisition unit 411 . The instruction signal includes an imaging start instruction signal. When the determination unit 418 determines that the signal acquisition unit 411 has acquired the imaging start command signal, the three-dimensional position calculation unit 417 performs stereo processing based on the image PC captured after the acquisition of the imaging start command signal. to calculate the three-dimensional position of the construction target SB.

The start time and stop time of photographing are determined by operating the input device 32, so that the driver can adjust the timing of the operation of the input device 32 when, for example, only a partial area of the construction target SB is desired to be photographed. , it is possible to photograph only the desired area, and to suppress unnecessary areas from being photographed.

In the present embodiment, the determination unit 418 may use a stop condition other than the first to third stop conditions as the photographing stop condition. Further, the determination unit 418 may combine the first to third stop conditions to set the photographing stop conditions. For example, when the second stop condition and the third stop condition are combined, when a predetermined time has elapsed since the swinging body 3 stopped swinging, and when the signal obtaining unit 411 has acquired a photographing stop command signal by operating the input device 32 . , may be used as conditions for stopping photography.

[Input device]
FIG. 9 is a perspective view schematically showing an example of the input device 32 according to this embodiment. In this embodiment, the measurement system 50 has a turning continuous photographing mode in which the imaging device 30 continuously photographs the construction target SB while the revolving structure 3 is turning, and a turning continuous photographing mode in which the construction target SB is continuously photographed by the imaging device 30 while the revolving structure 3 is not turning. A single image shooting mode for shooting SB can be implemented. The input device 32 includes a turning continuous photographing switch 32A capable of generating a photographing start command signal for commanding the start of the turning continuous photographing mode and a photographing stop command signal for instructing the stop of the turning continuous photographing mode, and a photographing in the single image photographing mode. and a single image shooting switch 32B capable of generating a command single image shooting command signal. In addition, a first light emitting section 34A is provided as part of the rotary continuous photographing switch 32A, and a second light emitting section 34B is provided as part of the single image photographing switch 32B. Each of the first light emitting section 34A and the second light emitting section 34B includes a light emitting diode. The first light emitting section 34A is lit, blinked, or extinguished based on the operating state of the turning continuous photographing switch 32A. The second light emitting section 34B lights, blinks, or goes out based on the operating state of the single image shooting switch 32B.

[Measurement method]
FIG. 10 is a flow chart showing an example of the measurement method according to this embodiment. FIG. 11 is a timing chart of the measurement method according to this embodiment. In the following description, the measurement method based on the second stop condition among the first, second, and third stop conditions will be described.

The operator of the hydraulic excavator 1 operates the operation device 35 to turn the revolving body 3 so that the imaging device 30 faces the measurement start position of the construction target SB. At time t1 after a predetermined time has elapsed from time t0 when the operation of the hydraulic excavator 1 including the revolving of the revolving body 3 is stopped, the determination unit 418 determines whether the position detection device 23 and the attitude detection device 24 are detected. is in a static state in which detection data can be stably output.

When the hydraulic excavator 1 is in the operation stop state and the position detection device 32 and the attitude detection device 24 are in the static state, when the imaging start command signal is acquired, the imaging position calculation unit 416 detects the position of the revolving body 3 from the position detection device 23. is obtained, and detection data indicating the attitude of the revolving body 3 is obtained from the attitude detection device 24 (step S10).

The imaging position calculator 416 acquires the pre-turning angle θra and the pre-turning position Pra.

The detection data of the position detection device 23 and the detection data of the orientation detection device 24 are temporarily stored in the storage device 42 .

When starting the turning continuous photographing mode, the operator of the hydraulic excavator 1 operates (presses) the turning continuous photographing switch 32A. In the example shown in FIG. 11, the turning continuous photographing switch 32A is operated at time t2. A shooting start command signal generated by operating the turning continuous shooting switch 32</b>A is output to the arithmetic processing unit 41 . The signal acquisition unit 411 acquires the imaging start command signal (step S20).

Further, when the turning continuous photographing switch 32A is operated and a photographing start command signal is acquired, the processing unit 41 starts photographing by the photographing device 30 (step S30). The image acquisition unit 410 acquires an image PCra of the construction target SB captured by the imaging device 30 while the hydraulic excavator 1 is in an operation stop state before starting operation. The image storage unit 423 also stores the image PCra.

The driver operates the operating device 35 to start the swinging of the swinging structure 3 while the traveling structure 5 is stopped (step S40). The driver turns the operation device 35 from the turn start position where the imaging device 30 faces the measurement start position of the construction target SB to the turn stop position where the imaging device 30 steps on the measurement stop position of the construction target SB. By operating it, the revolving body 3 starts revolving.

In the following description, the turning direction RD when turning from the turning start position toward the turning stop position is appropriately referred to as forward direction. In addition, the direction opposite to the forward direction in the turning direction RD is appropriately referred to as the reverse direction.

In the example shown in FIG. 11, the operating device 35 is operated at time t3, and the swinging body 3 starts swinging. Each of the plurality of imaging devices 30 (30A, 30B, 30C, 30D) captures images PCs of the construction target SB multiple times at intervals while the revolving body 3 is revolving.

The image acquisition unit 410 sequentially acquires a plurality of images PCs of the construction target SB captured by the imaging device 30 while the revolving body 3 is revolving. Also, the image storage unit 423 sequentially stores a plurality of image PCs.

In the present embodiment, after the turning continuous photographing switch 32A is operated and the photographing start command signal is acquired by the signal acquisition unit 411, the photographing device 30 continues until the photographing stop condition is satisfied. The image PC of the construction target SB is continuously photographed. In other words, during the period between the time when the imaging start command signal is acquired and the time when the imaging stop condition is satisfied, the imaging device 30 can be set to , continue to photograph the construction target SB.

When the revolving superstructure 3 reaches the revolving stop position, the driver releases the operation of the operation device 35 to stop revolving of the revolving superstructure 3 (step S60). In the example shown in FIG. 11, the operation of the operating device 35 is released at time t4, and the swinging body 3 stops swinging.

At time t5 after a lapse of a predetermined time from time t4 when the swinging body 3 stops turning, the determination unit 418 determines that the detection data of the position detection device 23 and the attitude detection device 24 have stabilized. It is determined that the output is in a static state.

When the hydraulic excavator 1 is in an operation stop state and the position detection device 32 and the attitude detection device 24 are in a stationary state, the photographing position calculation unit 416 acquires detection data indicating the position of the revolving body 3 from the position detection device 23, Detection data indicating the attitude of the revolving body 3 is acquired from the attitude detection device 24 (step S70).

The imaging position calculator 416 acquires the post-turning angle θrb and the post-turning position Prb.

The detection data of the position detection device 23 and the detection data of the orientation detection device 24 are temporarily stored in the storage device 42 .

The image acquiring unit 410 also acquires the image PCrb of the construction target SB captured by the imaging device 30 in the operation stop state after the hydraulic excavator 1 stops operating. The image storage unit 423 also stores the image PCrb.

The determination unit 418 determines whether or not the operation data or the turning data of the operation device 35 satisfies the photographing stop condition from the time t4 when the turning of the turning body 3 is stopped to the time t6 after a predetermined time has passed. (step S80).

In step S80, it is determined whether or not a condition in which a predetermined time has elapsed since the revolving structure 3 has stopped revolving is satisfied as a condition for stopping photographing (second stop condition). When it is determined that the predetermined time has passed since the revolving body 3 stopped revolving, it is determined that the operation data or the revolving data satisfies the photographing stop condition.

As described above, if the operation data or turning data does not satisfy the photographing stop condition, when the turning body 3 starts turning again after stopping turning, the turning stop time from turning stop to turning restart is less than a predetermined time. The turning direction RD of the turning body 3 after the turning of the turning body 3 is restarted may be the forward direction or the reverse direction. The determination unit 418 determines that the operation data or the turning data satisfies the photographing stop condition when determining that the predetermined time has passed since the turning of the turning body 3 was stopped. On the other hand, the determination unit 418 determines whether the swing stop time from when the swing is stopped to when the swing is restarted when the predetermined time has not passed since the swing of the swing structure 3 was stopped, or when the swing is restarted after stopping the swing. is less than the specified time, it is determined that the operation data or the turning data does not satisfy the photographing stop condition.

When it is determined in step S80 that the operation data or turning data satisfies the photographing stop condition, that is, when it is determined that the predetermined time has not passed since the turning of the turning body 3 stopped (step S80: No ), the process of step S80 is performed until the stop condition is satisfied.

When it is determined in step S80 that the operation data or turning data satisfies the photographing stop condition, that is, when it is determined that a predetermined time has elapsed since the turning of the turning body 3 stopped (step S80: Yes), the imaging device 30 is stopped (step S90). The imaging device 30 transitions to the imaging disabled state.

The common part extraction unit 413 extracts the common part KS from each of the plurality of images PC stored in the image storage unit 423 (step S120).

The common part extraction unit 413 extracts a common part KS of at least two images PC captured by the imaging device 30 while the revolving body 3 is stopping and revolving. The extraction of the common portion KS is performed for all images PC acquired in the turning direction R of the turning body 3 from when the turning of the turning body 3 starts to when it stops.

As described above, the turning of the turning body 3 may stop while the turning body 3 is turning from the turning start position toward the turning stop position. The imaging device 30 continues to capture the image PC even during the rotation stop time from when the rotation of the rotating body 3 is stopped to when it is restarted. The common part extraction unit 413 extracts a common part KS between the image PC captured by the imaging device 30 during the rotation stop time and the image PC captured by the imaging device 30 immediately before the rotation is stopped. In addition, the common part extraction unit 413 extracts a common part KS between the image PC captured by the imaging device 30 during the rotation stop time and the image PC captured by the imaging device 30 immediately after the rotation is resumed. As described above, the revolving direction RD of the revolving superstructure 3 after resuming revolving may be forward or backward.

The common part extracting unit 413 extracts a common part KS of the images PC captured during the turning stop time in which the time from stopping to restarting turning is equal to or less than a specified time. That is, when the revolving body 3 stops while it is revolving from the revolving start position toward the revolving stop position, the common portion KS is extracted for the image PC captured during the revolving stop time equal to or shorter than the specified time. , the common portion KS is not extracted for the images PC captured during the rotation stop time exceeding the specified time.

Next, the imaging position calculator 416 estimates the estimated angle θs of the revolving body 3 based on the common portion KS of the multiple images PC (step S140). The imaging position calculator 416 can calculate the turning angle Δθ and estimate the estimated angle θs based on the positions of the common portions KS in the multiple images PC.

When the difference between the estimated angle θs after turning stop and the post-turning angle θrb is less than a specified value, the imaging position calculation unit 416 calculates the angle θr (pre-turning angle θra, post-turning angle θrb) during turning. Correct the estimated angle θs. Further, the imaging position calculation unit 416 corrects the estimated position Ps of the imaging device 30 during turning based on the corrected estimated angle θs. The imaging position calculation unit 416 can correct the estimated angle θs based on the procedure described above.

The three-dimensional position calculation unit 417 stereo-processes the image PC at the time of photographing, and calculates the three-dimensional positions of the plurality of construction targets SB in the imaging device coordinate system (step S160). Also, the three-dimensional position calculation unit 417 converts the three-dimensional position in the imaging device coordinate system into a three-dimensional position in the field coordinate system.

As described above, when the revolving body 3 stops revolving in the middle of revolving from the revolving start position toward the revolving stop position, the image PC captured before the revolving stop time is equal to or less than the specified time. The captured image PC and the image PC captured after the resumption of turning are captured by the imaging device 30 . The three-dimensional position calculation unit 417 can perform stereo processing on the images PC captured before stopping turning, during stopping turning, and after restarting turning, and can calculate the three-dimensional position of the construction target SB. .

[effect]
As described above, according to the present embodiment, the stop time point for stopping the photographing of the image PC used for stereo processing is determined based on the operation data of the operating device 35 or the turning data of the turning body 3 . As a result, the area of the construction target SB whose terrain is to be measured can be efficiently photographed by stereo processing, and the three-dimensional shape of the construction target SB can be obtained smoothly.

Further, in the present embodiment, when the swinging body 3 is stopped while swinging from the swinging start position to the swinging stop position, whether or not the swinging stop time from stopping to restarting swinging is equal to or less than a predetermined time. is determined. When it is determined that the turning stop time is less than or equal to the predetermined time, the three-dimensional position of the construction target SB is calculated based on the image PC captured before and after turning is stopped and after the turning is restarted, respectively, and the position of the imaging device 30. be done. Further, in the present embodiment, even if the rotation of the rotating body 3 is stopped while it is rotating from the rotation start position to the rotation stop position, after the shooting start command signal is acquired by operating the rotation continuous shooting switch 32A, During the period until the photographing stop condition is satisfied, the imaging device 30 continues photographing the construction target SB even if the revolving body 3 stops revolving. As a result, it is possible to prevent photography from stopping every time the revolving body 3 stops revolving, or when the revolving body 3 stops revolving for less than a predetermined period of time. Therefore, a decrease in processing efficiency when measuring the three-dimensional shape of the construction target SB is suppressed.

An operator of the hydraulic excavator 1 may operate the operation device 30 to intentionally stop the swing of the swing body 3 . Further, the turning of the turning body 3 may be stopped without the driver's intention. In such a case, if the photographing is stopped every time the revolving body 3 stops rotating, or if the image PC photographed while the revolving is stopped is not used for stereo processing, the driver will have to restart the photographing. must be implemented. In this embodiment, even if the revolving body 3 stops revolving, if the revolving is restarted within a predetermined time, the photographing is continued. Therefore, a decrease in processing efficiency when measuring the three-dimensional shape of the construction target SB is suppressed.

Further, when photographing is to be performed while rotating the revolving body 3, the operator of the hydraulic excavator 1 stops revolving the revolving body 3, then revolves the revolving body 3 in the opposite direction, and shoots the work object for which photography has already stopped. SB may be shot again. In such a case, if the photographing is stopped every time the swing body 3 swings in the opposite direction, or if the image PC photographed during the swing in the reverse direction is not used for stereo processing, the tertiary The processing efficiency when measuring the original shape is lowered. In the present embodiment, even when the revolving structure 3 that has been revolving in the forward direction is revolved in the reverse direction, for example, for redoing imaging, continuous revolving imaging is continued as long as the revolving stop time is less than the predetermined time. be. Therefore, a decrease in processing efficiency when measuring the three-dimensional shape of the construction target SB is suppressed.

Further, as in the third stop condition, the stop time point for stopping the shooting of the image PC used for the stereo processing is determined based on the shooting stop command signal generated by operating the input device 32. good too. In this case as well, the area of the construction target SB whose terrain is to be measured can be efficiently photographed by stereo processing, and the three-dimensional shape of the construction target SB can be obtained smoothly.

In the above-described embodiment, the period from time t4 when turning is stopped to time t5 of the determination of static stability is shorter than time t6 after a predetermined time has elapsed from time t4, but the period from time t4 to time 5 may be longer than or equal to the period from time t4 to time t6.

In the above-described embodiment, while the hydraulic excavator 1 is turning, the position of the imaging device 30 is calculated based on the common portion KS, but the present invention is not limited to this embodiment. For example, the position of the imaging device 30 while the hydraulic excavator 1 is turning may be calculated based on the detection data of the position detection device 23 and the detection data of the attitude detection device 24 .

In the above-described embodiment, the imaging position calculation unit 416 uses the turning axis Zr as a constraint condition and calculates the imaging position using only the turning angle θ as a variable. The position and attitude of 30 may be calculated based on six variables: X-axis position, Y-axis position, Z-axis position, roll angle, pitch angle, and yaw angle.

In another embodiment, the imaging device 30 may start or stop imaging by operating the operation device 35 . When the operating device 35 is put into an operating state to start the swinging of the swinging body 3, the imaging device 30 starts photographing based on the operation data from which the operating state is detected. When the operating device 35 is in the neutral state, the imaging device 30 may stop photographing based on the operation data for detecting the neutral state. Alternatively, shooting may be started by operating the input device 32 and stopped when the operating device 35 is in a neutral state. Further, the determination as to whether or not the revolving body 3 is currently revolving may be performed by an arbitrary turning detection method without being based on the operation data detected by the operation amount sensor 36 .

Further, when the input device 32 is operated, the photographing by the imaging device 30 and the turning of the revolving body 3 are automatically started. The turning of the body 3 may be stopped.

In the above-described embodiment, the input device 32 may be attached to, for example, at least one of the right operating lever 35R and the left operating lever 35L of the operating device 35, or may be provided on a monitor panel arranged in the operator's cab 4. It may be provided in the portable terminal device. Further, the input device 32 may be provided outside the hydraulic excavator 1 to remotely control the start or stop of imaging by the imaging device 30 .

Note that when the image PC captured by the imaging device 30 is accumulated in the image storage unit 423, the condition that the data of the image PC of a specified capacity is accumulated (memory over condition) or the image is A condition (time over condition) may be set that the time for which the data of the PC is accumulated exceeds a specified time.

In the above-described embodiment, the imaging position calculation unit 416 calculates the turning data based on the common portion KS, and the turning data acquisition unit 415 acquires the turning data from the imaging position calculation unit 416. The turning data acquisition unit 415 may calculate turning data based on detection data of a detection device for calculating turning data. For example, the turning data acquisition unit 415 calculates turning data including the turning angle θ, turning speed V, and turning direction R of the turning body 3 based on the detection data of the position detection device 23 or the detection data of the posture detection device 24. Alternatively, the turning data may be calculated based on the operation data of the operating device 35 detected by the operation amount sensor 36, or the turning data of the turning body 3 may be detected by an angle sensor such as a rotary encoder. Turn data may be calculated based on the detected data. In this case, the arithmetic processing unit 41 synchronizes the timing of acquiring the turning angle θ, which is the detection value of the angle detection sensor, from the angle detection sensor and the timing of at least a pair of imaging devices 30 photographing the construction target SB. In this way, the timing at which the image PC is captured by at least one pair of imaging devices 30 is associated with the turning angle θ of the turning body 3 at that timing.

In the above-described embodiment, the arithmetic processing unit 41 stereo-processes the images PC captured by at least a pair of imaging devices 30 to achieve three-dimensional measurement, but the present invention is not limited to this. For example, an image PC of the excavation target SB around the hydraulic excavator 1 captured by at least a pair of imaging devices 30, and the position and posture of the hydraulic excavator 1 at rest obtained by the position detection device 23 and the posture detection device 24. is transmitted to, for example, a management device (portable terminal, server device, etc.) external to the hydraulic excavator 1 . Then, an external management device stereo-processes the image PC of the excavation target SB around the hydraulic excavator 1, and obtains the turning angle θ during turning of the turning body 3 and the position and attitude of the hydraulic excavator 1, and obtains the results. may be used to determine the three-dimensional position of the excavation object SB around the hydraulic excavator 1 during turning. In this case, the management device outside the hydraulic excavator 1 corresponds to the arithmetic processing device 41 . That is, in other embodiments, the measurement system 50 does not need to implement all functions in the hydraulic excavator 1 alone, and even if an external management device has some functions, for example, the functions of the arithmetic processing unit 41, good.

In the embodiments described above, the imaging device is a stereo camera including at least a pair of imaging devices 30 . When photographing with a stereo camera, the photographing timing of each camera is synchronized. The imaging device may be capable of stereo imaging with one camera. That is, it may be an imaging apparatus capable of stereo processing based on two images captured at different timings by one camera. An imaging device is not limited to a stereo camera. The imaging device may be, for example, a sensor capable of obtaining both an image and three-dimensional data, such as a TOF (Time Of Flight) camera. The imaging device may be an imaging device that obtains three-dimensional data with one camera. The imaging device may be a laser scanner.

In the above-described embodiment, the working machine 1 is the hydraulic excavator 1 having the revolving body 3 . The working machine 1 may be a working machine that does not have a revolving body. For example, the work machine may be at least one of a bulldozer, wheel loader, dump truck, and motor grader.

DESCRIPTION OF SYMBOLS 1... Hydraulic excavator (working machine), 1B... Vehicle body, 2... Working machine, 3... Revolving body, 4... Driver's cab, 4S... Driver's seat, 5... Running body, 5A, 5B... Crawler, 6... Boom, 7... Arm 8 Bucket 9 Counterweight 10 Boom cylinder 11 Arm cylinder 12 Bucket cylinder 21 GPS antenna 23 Position detection device 24 Attitude detection device 30 Imaging device 30A, Reference numerals 30B, 30C, 30D: Imaging device 31: Hub 32: Input device 32A: Rotating continuous photographing switch 32B: Single image photographing switch 35: Operating device 35L: Left operating lever 35R: Right operating lever 36 Operation amount sensor 37 Turning sensor 40 Control device 41 Arithmetic processing device 42 Storage device 43 Input/output interface 50 Measurement system 300 Stereo camera 301 First stereo camera 302 Second stereo camera 410 Image acquisition unit 411 Signal acquisition unit 413 Common part extraction unit 415 Turn data acquisition unit 416 Imaging position calculation unit 417 Three-dimensional position calculation unit 418 Determination Section 419 Output section 422 Stop condition storage section 423 Image storage section FM Shooting area KS Common portion MD Running direction PC Image PK Calculation image Pr Turning position Pra... position before turning, Prb... position after turning, Ps... estimated position, RD... turning direction, SB... construction target, WD... extraction construction target area, WD1, WD2, WD3, WD4... division area, WE... non-construction target area, Zr: turning axis, θr: turning angle, θra: pre-turning angle, θrb: post-turning angle, θs: estimated angle.

Claims (5)

an image acquisition unit that acquires an image of a construction target captured by an imaging device mounted on a revolving body of a work machine while the revolving body is revolving;
a determination unit that determines whether or not the photographing stop condition is satisfied based on the operation data of the operating device that operates the revolving body or the revolving data of the revolving body;
an output unit that outputs a shooting stop command signal based on the determination result of the determination unit;
a three-dimensional position calculation unit that calculates the three-dimensional position of the construction target based on the image acquired by the image acquisition unit;
The three-dimensional position calculation unit calculates, when the revolving body is revolved within a predetermined time after the revolving body stops revolving, based on the images taken before revolving is stopped and after resuming revolving. to calculate the three-dimensional position;
Measuring system for working machines. The turning direction of the re-turning is opposite to the turning direction before the turning stop.
The measuring system for a working machine according to claim 1. further comprising an input device operated to stop imaging by the imaging device,
The determination unit determines whether or not the shooting stop condition is satisfied based on the operation of the input device.
A measuring system for a working machine according to claim 1 or 2. A working machine comprising the measuring system for a working machine according to any one of claims 1 to 3. Acquiring an image of a construction target taken during revolving of the revolving body by an imaging device mounted on the revolving body of the working machine;
Determining whether or not a condition for stopping photographing by the imaging device is satisfied based on operation data of an operating device that operates the revolving body or revolving data of the revolving body;
outputting a shooting stop command signal based on the result of the determination;
calculating the three-dimensional position of the construction target based on the acquired image;
When the revolving structure is re-swinged within a predetermined time after the revolving structure stops revolving, the three-dimensional position is calculated based on the images taken before revolving stops and after resuming revolving. do,
How to measure working machines.

Images