JP6895350B2 - Work machine measurement system and work machine - Google Patents

Description

The present invention relates to a work machine measurement system, a work machine, and a work machine measurement method.

In the technical field related to work machines, work machines equipped with an imaging device as disclosed in Patent Document 1 are known. By stereo-processing the pair of images taken by the pair of image pickup devices, the three-dimensional shape of the construction target around the work machine is measured.

International Publication No. 2017/033991

When a construction object around a work machine is continuously photographed by an imaging device while rotating a rotating body, the position of the imaging device may be acquired based on the captured image. For example, when the position of the image pickup device is acquired based on the feature points common to a plurality of images, it is difficult to accurately acquire the position of the image pickup device depending on the extraction state of the feature points. As a result, the measurement accuracy of the three-dimensional shape of the target may decrease.

An aspect of the present invention is to suppress a decrease in measurement accuracy of a three-dimensional shape of an object.

According to the aspect of the present invention, an image acquisition unit that acquires a plurality of images of a construction target taken by an image pickup device mounted on a swivel body during operation of a work machine and a common portion in the plurality of images are extracted. A common part extraction unit that divides the extraction target area into division areas and extracts the common part from each of the division areas, and position data of the work machine detected by the position detection unit that detects the position of the work machine. An imaging position for calculating the position and orientation of the imaging device at the time when the image is captured, based on the attitude data of the work machine detected by the attitude detection unit that detects the attitude of the work machine and the common portion. A measurement system for a work machine including a calculation unit, a three-dimensional position calculation unit that calculates a three-dimensional position of the construction target based on the position data, the posture data, and the image is provided.

According to the aspect of the present invention, it is possible to suppress a decrease in measurement accuracy of the three-dimensional shape of the object.

FIG. 1 is a perspective view showing an example of a work machine according to the present embodiment. FIG. 2 is a perspective view showing a part of the work machine according to the present embodiment. FIG. 3 is a functional block diagram showing an example of the measurement system according to the present embodiment. FIG. 4 is a diagram schematically showing an example of the operation of the work machine according to the present embodiment. FIG. 5 is a diagram schematically showing an example of the operation of the measurement system according to the present embodiment. FIG. 6 is a schematic diagram for explaining an example of processing of the measurement system according to the present embodiment. FIG. 7 is a schematic diagram for explaining an example of processing of the measurement system according to the present embodiment. FIG. 8 is a schematic diagram for explaining an example of processing of the measurement system according to the present embodiment. FIG. 9 is a diagram schematically showing a state in which one extraction target area is set in the image. FIG. 10 is a diagram showing an example of an image extraction target region according to the present embodiment. FIG. 11 is a diagram in which common portions are extracted in an image in which the contrast distribution of each divided region is corrected by the histogram flattening algorithm. It is a schematic diagram for demonstrating the histogram flattening algorithm. FIG. 12 is a flowchart showing an example of the measurement method according to the present embodiment. FIG. 13 is a timing chart of the measurement method according to the present embodiment. FIG. 14 is a schematic diagram for explaining an example of processing of the measurement system according to the present embodiment.

Hereinafter, embodiments according to the present invention will be described with reference to the drawings, but the present invention is not limited thereto. The components of the embodiments described below can be combined as appropriate. In addition, some components may not be used.

In the following description, the three-dimensional field coordinate system (Xg, Yg, Zg), the three-dimensional vehicle body coordinate system (Xm, Ym, Zm), and the three-dimensional image pickup device coordinate system (Xs, Ys, Zs) are referred to. The positional relationship of each part will be described by definition.

The field coordinate system is a coordinate system based on the origin fixed to the earth. The field coordinate system is a coordinate system defined by GNSS (Global Navigation Satellite System). GNSS refers to a global navigation satellite system. GPS (Global Positioning System) is an example of a global navigation satellite system.

The field coordinate system is defined by the Xg axis of the horizontal plane, the Yg axis of the horizontal plane orthogonal to the Xg axis, and the Xg axis and the Zg axis orthogonal to the Yg axis. The rotation or tilt direction centered on the Xg axis is defined as the θXg direction, the rotation or tilt direction centered on the Yg axis is defined as the θYg direction, and the rotation or tilt direction centered on the Zg axis is defined as the θZg direction. The Zg axis direction is the vertical direction.

The vehicle body coordinate system consists of the Xm axis of the first predetermined surface based on the origin defined on the vehicle body of the work machine, the Ym axis of the first predetermined surface orthogonal to the Xm axis, and the Xm axis and Zm orthogonal to the Ym axis. Defined by the axis. The rotation or tilt direction centered on the Xm axis is defined as the θXm direction, the rotation or tilt direction centered on the Ym axis is defined as the θYm direction, and the rotation or tilt direction centered on the Zm axis is defined as the θZm direction. The Xm axis direction is the front-rear direction of the work machine, the Ym axis direction is the vehicle width direction of the work machine, and the Zm axis direction is the vertical direction of the work machine.

The image pickup device coordinate system includes the Xs axis of the second predetermined plane with reference to the origin defined in the image pickup device, the Ys axis of the second predetermined plane orthogonal to the Xs axis, and the Xs axis and the Zs axis orthogonal to the Ys axis. Is defined by. The rotation or tilt direction centered on the Xs axis is defined as the θXs direction, the rotation or tilt direction centered on the Ys axis is defined as the θYs direction, and the rotation or tilt direction centered on the Zs axis is defined as the θZs direction. The Xs-axis direction is the vertical direction of the image pickup device, the Ys-axis direction is the width direction of the image pickup device, and the Zs-axis direction is the front-back direction of the image pickup device. The Zs axis direction is parallel to the optical axis of the optical system of the image pickup apparatus.

The position in the field coordinate system, the position in the vehicle body coordinate system, and the position in the image pickup device coordinate system can be converted to each other.

First embodiment.
[Working machine]
FIG. 1 is a perspective view showing an example of the work machine 1 according to the present embodiment. In this embodiment, an example in which the work machine 1 is a hydraulic excavator will be described. In the following description, the work machine 1 is appropriately referred to as a hydraulic excavator 1.

As shown in FIG. 1, the hydraulic excavator 1 has a vehicle body 1B and a working machine 2. The vehicle body 1B has a swivel body 3 and a traveling body 5 that supports the swivel body 3 so as to be swivelable.

The swivel body 3 has a driver's cab 4. The hydraulic pump and the internal combustion engine are arranged in the swivel body 3. The swivel body 3 can swivel around the swivel shaft Zr. The turning axis Zr is parallel to the Zm axis of the vehicle body coordinate system. The origin of the vehicle body coordinate system is defined, for example, at the center of the swing circle of the swivel body 3. The center of the swing circle is located on the swivel axis Zr of the swivel body 3.

The traveling body 5 has tracks 5A and 5B. The hydraulic excavator 1 travels by rotating the tracks 5A and 5B. The Zm axis of the vehicle body coordinate system is orthogonal to the ground plane of the tracks 5A and 5B. The upper part of the vehicle body coordinate system (+ Zm direction) is the direction away from the ground plane of the tracks 5A and 5B, and the lower part of the vehicle body coordinate system (−Zm direction) is the direction opposite to the upper part of the vehicle body coordinate system.

The working machine 2 is connected to the swivel body 3. In the vehicle body coordinate system, at least a part of the working machine 2 is arranged in front of the swivel body 3. The front of the vehicle body coordinate system (+ Xm direction) is the direction in which the work machine 2 exists with reference to the swivel body 3, and the rear of the vehicle body coordinate system (-Xm direction) is the direction opposite to the front of the vehicle body coordinate system. is there.

The work machine 2 drives the boom 6 connected to the swivel body 3, the arm 7 connected to the boom 6, the bucket 8 connected to the arm 7, the boom cylinder 10 for driving the boom 6, and the arm 7. It has an arm cylinder 11 and a bucket cylinder 12 for driving the bucket 8. The boom cylinder 10, arm cylinder 11, and bucket cylinder 12 are hydraulic cylinders that are driven by flood control, respectively.

Further, the hydraulic excavator 1 includes a position detection device 23 for detecting the position of the swivel body 3, a posture detection device 24 for detecting the posture of the swivel body 3, and a control device 40.

The position detection device 23 detects the position of the swivel body 3 in the field coordinate system. The position detection device 23 functions as a position detection unit that detects the position of the hydraulic excavator 1. The position of the swivel body 3 includes the coordinates in the Xg axis direction, the coordinates in the Yg axis direction, and the coordinates in the Zg axis direction. The position detection device 23 includes a GPS receiver. The position detection device 23 is provided on the swivel body 3.

The GPS antenna 21 is provided on the swivel body 3. Two GPS antennas 21 are arranged, for example, in the Ym axis direction of the vehicle body coordinate system. The GPS antenna 21 receives radio waves from GPS satellites and outputs a signal generated based on the received radio waves to the position detection device 23. The position detection device 23 detects the position of the GPS antenna 21 in the field coordinate system based on the signal from the GPS antenna 21.

The position detection device 23 performs arithmetic processing based on at least one of the positions of the two GPS antennas 21 to calculate the position of the swivel body 3. The position of the swivel body 3 may be the position of one GPS antenna 21 or the position between the position of one GPS antenna 21 and the position of the other GPS antenna 21.

The posture detection device 24 detects the posture of the swivel body 3 in the field coordinate system. The posture detection device 24 functions as a posture detection unit that detects the posture of the hydraulic excavator 1. The postures of the swivel body 3 are a roll angle indicating the tilt angle of the swivel body 3 in the rotation direction centered on the Xm axis, a pitch angle indicating the tilt angle of the swivel body 3 in the rotation direction centered on the Ym axis, and Zm. Includes an azimuth angle indicating the inclination angle of the swivel body 3 in the rotation direction centered on the axis. The attitude detection device 24 includes an inertial measurement unit (IMU). The posture detection device 24 is provided on the swivel body 3. A gyro sensor may be mounted on the swivel body 3 as the posture detection device 24.

The attitude detection device 24 detects the acceleration and the angular velocity acting on the attitude detection device 24. By detecting the acceleration and the angular velocity acting on the attitude detection device 24, the acceleration and the angular velocity acting on the swivel body 3 are detected. The posture detection device 24 performs arithmetic processing based on the acceleration and the angular velocity acting on the swivel body 3 to calculate the posture of the swivel body 3 including the roll angle, the pitch angle, and the azimuth angle.

The azimuth may be calculated based on the detection data of the position detection device 23. The position detection device 23 can calculate the azimuth angle of the swivel body 3 with respect to the reference direction in the field coordinate system based on the position of one GPS antenna 21 and the position of the other GPS antenna 21. The reference direction is, for example, north. The position detection device 23 calculates a straight line connecting the position of one GPS antenna 21 and the position of the other GPS antenna 21, and based on the angle formed by the calculated straight line and the reference direction, the swivel body 3 with respect to the reference direction The azimuth angle can be calculated.

Next, the stereo camera 300 according to the present embodiment will be described. FIG. 2 is a perspective view showing a part of the hydraulic excavator 1 according to the present embodiment. As shown in FIG. 2, the hydraulic excavator 1 has a stereo camera 300. The stereo camera 300 refers to a camera capable of measuring the distance to the construction target SB by simultaneously photographing the construction target SB from a plurality of directions and generating parallax data.

The stereo camera 300 photographs the construction target SB around the hydraulic excavator 1. The construction target SB includes an excavation target excavated by the work machine 2 of the hydraulic excavator 1. The construction target SB may be a construction target constructed by a work machine different from the hydraulic excavator 1, or may be a construction target constructed by an operator. Further, the construction target SB is a concept including a construction target before construction, a construction target during construction, and a construction target after construction.

The stereo camera 300 is mounted on the swivel body 3. The stereo camera 300 is provided in the driver's cab 4. The stereo camera 300 is arranged in front of (+ Xm direction) and above (+ Zm direction) of the driver's cab 4. The stereo camera 300 photographs the construction target SB in front of the hydraulic excavator 1.

The stereo camera 300 has a plurality of image pickup devices 30. The image pickup device 30 is mounted on the swivel body 3. The image pickup apparatus 30 has an optical system and an image sensor. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. In the present embodiment, the image pickup device 30 includes four image pickup devices 30A, 30B, 30C, and 30D.

The stereo camera 300 is composed of a pair of image pickup devices 30. The stereo camera 300 includes a first stereo camera 301 composed of a pair of image pickup devices 30A and 30B, and a second stereo camera 302 composed of a pair of image pickup devices 30C and 30D.

The image pickup devices 30A and 30C are arranged on the + Ym side (working machine 2 side) of the image pickup devices 30B and 30D. The image pickup apparatus 30A and the image pickup apparatus 30B are arranged at intervals in the Ym axis direction. The image pickup apparatus 30C and the image pickup apparatus 30D are arranged at intervals in the Ym axis direction. The image pickup devices 30A and 30B are arranged on the + Zm side of the image pickup devices 30C and 30D. In the Zm axis direction, the image pickup device 30A and the image pickup device 30B are arranged at substantially the same position. In the Zm axis direction, the image pickup apparatus 30C and the image pickup apparatus 30D are arranged at substantially the same position.

The image pickup devices 30A and 30B face upward (+ Zm direction). The image pickup devices 30C and 30D face downward (-Zm direction). Further, the image pickup devices 30A and 30C face forward (+ Xm direction). The image pickup devices 30B and 30D face the + Ym side (working machine 2 side) slightly from the front. That is, the image pickup devices 30A and 30C face the front surface of the swivel body 3, and the image pickup devices 30B and 30D face the image pickup devices 30A and 30C. The image pickup devices 30B and 30D may face the front surface of the swivel body 3, and the image pickup devices 30A and 30C may face the image pickup devices 30B and 30D.

The image pickup apparatus 30 photographs the construction target SB existing in front of the swivel body 3. By stereoprocessing the pair of images taken by the pair of imaging devices 30 in the control device 40, three-dimensional data indicating the three-dimensional shape of the construction target SB is calculated. The control device 40 converts the three-dimensional data of the construction target SB in the image pickup device coordinate system into the three-dimensional data of the construction target SB in the site coordinate system. The three-dimensional data indicates the three-dimensional position of the construction target SB. The three-dimensional position of the construction target SB includes the three-dimensional coordinates of each of the plurality of parts on the surface of the construction target SB.

An image pickup device coordinate system is defined for each of the plurality of image pickup devices 30. The image pickup device coordinate system is a coordinate system based on the origin fixed to the image pickup device 30. The Zs axis of the image pickup device coordinate system coincides with the optical axis of the optical system of the image pickup device 30.

In the present embodiment, two sets of stereo cameras (first stereo camera 301 and second stereo camera 302) are mounted on the swivel body 3, but one set of stereo cameras may be mounted. However, three or more sets of stereo cameras may be mounted.

Further, as shown in FIG. 2, the hydraulic excavator 1 has a driver's seat 4S, an input device 32, and an operation device 35. The driver's seat 4S, the input device 32, and the operation device 35 are arranged in the driver's cab 4. The driver of the hydraulic excavator 1 is seated in the driver's seat 4S.

The input device 32 is operated by the driver to start or end the shooting by the image pickup device 30. The input device 32 is provided in the vicinity of the driver's seat 4S. By operating the input device 32, shooting by the image pickup device 30 starts or ends.

The operating device 35 is operated by the driver for driving or stopping the driving of the work machine 2, turning or stopping the turning body 3, and running or stopping the running body 5. The operating device 35 includes a right operating lever 35R and a left operating lever 35L for operating the working machine 2 and the swivel body 3. Further, the operating device 35 includes a right traveling lever and a left traveling lever (not shown) for operating the traveling body 5. By operating the operating device 35, the working machine 2 is driven or the drive is stopped, the swivel body 3 is swiveled or swiveled and stopped, and the traveling body 5 is driven or stopped.

[Measurement system]
Next, the measurement system 50 according to the present embodiment will be described. FIG. 3 is a functional block diagram showing an example of the measurement system 50 according to the present embodiment. The measurement system 50 is provided on the hydraulic excavator 1.

The measurement system 50 is an operation for detecting the operation amount of the control device 40, the stereo camera 300 including the first stereo camera 301 and the second stereo camera 302, the position detection device 23, the attitude detection device 24, and the operation device 35. It includes an amount sensor 36 and an input device 32.

The control device 40 is provided on the swivel body 3 of the hydraulic excavator 1. The control device 40 includes a computer system. The control device 40 includes an arithmetic processing device 41 including a processor such as a CPU (Central Processing Unit), a volatile memory such as RAM (Random Access Memory), and a non-volatile memory such as ROM (Read Only Memory). It has a storage device 42 and an input / output interface 43.

The arithmetic processing unit 41 includes an image acquisition unit 410, a signal acquisition unit 411, an image processing unit 412, an intersection extraction unit 413, an imaging position calculation unit 416, a three-dimensional position calculation unit 417, and a determination unit 418. Has.

The image acquisition unit 410 acquires a plurality of image PCs of the construction target SB taken by the image pickup device 30 mounted on the swivel body 3 during the operation of the hydraulic excavator 1. Further, the image acquisition unit 410 acquires an image PC of the construction target SB photographed by the image pickup apparatus 30 in a state where the hydraulic excavator 1 is stopped, that is, when both traveling and turning are stopped.

The operation of the hydraulic excavator 1 includes one or both of the swivel body 3 being swiveled and the traveling body 5 being running. The fact that the hydraulic excavator 1 is stopped includes the fact that the turning body 3 is stopped turning and the traveling body 5 is stopped running.

The image acquisition unit 410 acquires a plurality of image PCs of the target SB taken by the image pickup apparatus 30 while the traveling body 5 is stopped and the rotating body 3 is turning. Further, the image acquisition unit 410 acquires an image PC of the target SB taken by the image pickup apparatus 30 while the traveling body 5 is stopped running and the turning body 3 is stopped turning.

The signal acquisition unit 411 acquires a command signal generated by operating the input device 32. The input device 32 is operated for the start or end of shooting by the image pickup device 30. The command signal includes a shooting start command signal and a shooting end command signal. The arithmetic processing unit 41 outputs a control signal for causing the imaging device 30 to start shooting based on the shooting start command signal acquired by the signal acquisition unit 411. Further, the arithmetic processing unit 41 outputs a control signal for ending the shooting to the imaging device 30 based on the shooting end command signal acquired by the signal acquisition unit 411.

The image PC captured during the period between the time when the signal acquisition unit 411 acquires the shooting start command signal and the time when the shooting end command signal is acquired is stored in the storage device 42, and is stored in the storage device 42. The image PC may be used for stereo processing.

The image processing unit 412 corrects the contrast distribution of the image PC acquired by the image acquisition unit 410. The image processing unit 412 corrects the contrast distribution of the image PC based on the histogram equalization algorithm, which is a kind of histogram correction method. Histogram flattening refers to a process of converting an image PC so that a histogram showing the relationship between the gradation and frequency of each pixel of the image PC is evenly distributed over the entire gradation. The image processing unit 412 performs image processing based on the histogram flattening algorithm to generate an image PC with improved contrast. The image processing unit 412 corrects the brightness (Luminance) of each pixel of the image PC to improve the contrast of the image PC.

The intersection extraction unit 413 extracts the intersection KS of a plurality of image PCs taken by the image pickup apparatus 30 while the excavator 3 of the hydraulic excavator 1 is rotating. The intersection KS will be described later.

The image pickup position calculation unit 416 calculates the position P and the posture of the image pickup apparatus 30 at the time of shooting. When the swivel body 3 is swirling, the position P of the image pickup device 30 at the time of photographing includes the position of the image pickup device 30 in the swivel direction RD. When the traveling body 5 is in the traveling state, the position P of the imaging device 30 at the time of shooting includes the position of the imaging device 30 in the traveling direction MD. Further, the imaging position calculation unit 416 calculates the turning angle θ of the turning body 3. The imaging position calculation unit 416 includes the position data of the hydraulic excavator 1 detected by the position detection device 23, the attitude data of the hydraulic excavator 1 detected by the attitude detection device 24, and the common portion KS extracted by the common portion extraction unit 413. Based on the above, the position P and the posture of the photographing device 30 at the time when the image PC is photographed are calculated.

The three-dimensional position calculation unit 417 calculates the three-dimensional position of the construction target SB in the image pickup device coordinate system by stereo-processing the pair of image PCs taken by the pair of image pickup devices 30. The three-dimensional position calculation unit 417 sets the three-dimensional position of the construction target SB in the image pickup device coordinate system to the construction target SB in the site coordinate system based on the position P and the orientation of the image pickup device 30 calculated by the image pickup position calculation unit 416. Convert to a three-dimensional position.

The storage device 42 has an image storage unit 423. The image storage unit 423 sequentially stores a plurality of image PCs captured by the image pickup device 30.

The input / output interface 43 includes an arithmetic processing unit 41 and an interface circuit for connecting the storage device 42 and an external device. A hub 31, a position detection device 23, an attitude detection device 24, an operation amount sensor 36, and an input device 32 are connected to the input / output interface 43.

The plurality of image pickup devices 30 (30A, 30B, 30C, 30D) are connected to the arithmetic processing unit 41 via the hub 31. The image pickup apparatus 30 takes an image PC of the construction target SB based on the shooting start command signal from the signal acquisition unit 411. The image PC of the construction target SB taken by the image pickup device 30 is input to each of the arithmetic processing unit 41 and the storage device 42 via the hub 31 and the input / output interface 43. Each of the image acquisition unit 410 and the image storage unit 423 acquires the image PC of the construction target SB taken by the image pickup apparatus 30 via the hub 31 and the input / output interface 43. The hub 31 may be omitted.

The input device 32 is operated for the start or end of shooting by the image pickup device 30. By operating the input device 32, a shooting start command signal or a shooting end command signal is generated. As the input device 32, at least one of an operation switch, an operation button, a touch panel, a voice input, and a keyboard is exemplified.

[Swirl terrain measurement]
Next, an example of the operation of the hydraulic excavator 1 according to the present embodiment will be described. FIG. 4 is a diagram schematically showing an example of the operation of the hydraulic excavator 1 according to the present embodiment. The measurement system 50 continuously photographs the image PC of the construction target SB around the hydraulic excavator 1 by the image pickup device 30 while the swivel body 3 is swiveling. While the swivel body 3 is swirling, the image pickup apparatus 30 sequentially captures image PCs of the construction target SB at a predetermined cycle.

The image pickup device 30 is mounted on the swivel body 3. As the swivel body 3 swivels, the imaging region FM of the image pickup apparatus 30 moves in the swivel direction RD. When the swivel body 3 is turning, the image pickup device 30 continuously shoots the image PCs of the construction target SB, so that the image pickup device 30 can acquire the image PCs of the plurality of regions of the construction target SB. The three-dimensional position calculation unit 417 can calculate the three-dimensional position of the construction target SB around the hydraulic excavator 1 by stereo-processing the pair of image PCs taken by the pair of image pickup devices 30.

The three-dimensional position of the construction target SB calculated by the stereo processing is defined in the image pickup device coordinate system. The three-dimensional position calculation unit 417 converts the three-dimensional position in the image pickup apparatus coordinate system into the three-dimensional position in the field coordinate system. In order to convert the three-dimensional position in the image pickup device coordinate system into the three-dimensional position in the field coordinate system, the position and orientation of the swivel body 3 in the field coordinate system are required. The position and orientation of the swivel body 3 in the field coordinate system can be detected by the position detection device 23 and the attitude detection device 24.

While the hydraulic excavator 1 is turning, each of the position detecting device 23 and the posture detecting device 24 mounted on the hydraulic excavator 1 is displaced. The detection data output from each of the position detection device 23 and the posture detection device 24 in the moving state may be unstable or the detection accuracy may be lowered.

The position detection device 23 outputs the detection data at a predetermined cycle. Therefore, when the position detection device 23 and the image pickup device 30 perform the position detection in parallel while the hydraulic excavator 1 is turning, the image pickup device 30 captures an image when the position detection device 23 is moving. There is a possibility that the timing at which the position is detected and the timing at which the position detection device 23 detects the position are not synchronized. If the three-dimensional position of the construction target SB is coordinate-converted based on the detection data of the position detection device 23 detected at a timing different from the shooting timing, the measurement accuracy of the three-dimensional position may decrease.

In the present embodiment, the measurement system 50 calculates the position and posture of the swivel body 3 at the time when the image PC is taken by the image pickup apparatus 30 while the swivel body 3 is swirling based on the method described later with high accuracy. As a result, the measurement system 50 can calculate the three-dimensional position of the construction target SB in the site coordinate system with high accuracy.

The imaging position calculation unit 416 acquires the detection data of each of the position detection device 23 and the posture detection device 24 detected when the hydraulic excavator 1 is stopped. It is highly possible that the detection data detected by the position detection device 23 and the posture detection device 24 when the hydraulic excavator 1 is stopped is stable. The imaging position calculation unit 416 acquires the detection data detected in each of the operation stop states before the start of turning and after the end of turning of the turning body 3. The image pickup device 30 captures an image PC of the construction target SB in each operation stop state before the start of the turn and after the end of the turn of the swivel body 3. The imaging position calculation unit 416 is based on the detection data of the position detection device 23 and the posture detection device 24 acquired when the hydraulic excavator 1 is stopped, and the image PC is taken from the image PC when the hydraulic excavator 1 is stopped. The calculated three-dimensional position of the construction target SB in the image pickup device coordinate system is converted into the three-dimensional position in the site coordinate system.

On the other hand, when the image PC of the construction target SB is photographed by the image pickup device 30 while the swivel body 3 is turning, the image pickup position calculation unit 416 is based on the method described later, and the image pickup device 30 uses the swivel body 3 while the swivel body 3 is turning. The position and posture of the swivel body 3 at the time when the image PC is taken are calculated. The image pickup position calculation unit 416 converts the three-dimensional position in the image pickup apparatus coordinate system calculated from the photographed image PC into the three-dimensional position in the field coordinate system based on the calculated position and orientation of the swivel body 3.

FIG. 5 is a diagram schematically showing an example of the operation of the measurement system 50 according to the present embodiment. FIG. 5 is a schematic view for explaining that the image pickup apparatus 30 photographs the construction target SB while the swivel body 3 is swiveling.

In the following description, it is assumed that the traveling body 5 is in the traveling stopped state. As shown in FIG. 5, when the swivel body 3 swivels, the image pickup device 30 mounted on the swivel body 3 and the photographing region FM of the image pickup device 30 move in the swivel direction RD. The imaging region FM of the imaging device 30 is defined based on the field of view of the optical system of the imaging device 30. The image pickup apparatus 30 acquires an image PC of the construction target SB arranged in the photographing area FM. Due to the rotation of the rotating body 3, the photographing region FM of the imaging device 30 moves in the turning direction RD of the rotating body 3. The image pickup apparatus 30 photographs the image PCs of the construction target SBs that are sequentially arranged in the moving photographing area FM.

FIG. 5 shows an example in which the photographing area FM moves in the turning direction RD in the order of the photographing area FM1, the photographing area FM2, and the photographing area FM3 due to the turning of the rotating body 3. The photographing area FM1 is defined by the first position PJ1 in the turning direction RD. The photographing area FM2 is defined by the second position PJ2 in the turning direction RD. The photographing area FM3 is defined by the third position PJ3 in the turning direction RD. The second position PJ2 is a position swiveled from the first position PJ1 by a turning angle Δθ1. The third position PJ3 is a position swiveled from the second position PJ2 by a turning angle Δθ2. The image pickup apparatus 30 captures each of the image PC1 of the construction target SB arranged in the photographing area FM1, the image PC2 of the construction target SB arranged in the photographing area FM2, and the image PC3 of the construction target SB arranged in the photographing area FM3. Take a picture. The image PC1, the image PC2, and the image PC3 are images captured by the same image pickup device 30 (the image pickup device 30C in the example shown in FIG. 5).

The image pickup apparatus 30 shoots at a predetermined timing so that the overlapping region OB is provided in the adjacent shooting region FM. FIG. 5 shows an example in which the overlapping area OB1 is provided in the photographing area FM1 and the photographing area FM2, and the overlapping area OB2 is provided between the photographing area FM2 and the photographing area FM3. The overlapping area OB1 is an overlapping area OB in which the image PC1 and a part of the image PC2 overlap. The overlapping area OB2 is an overlapping area OB in which the image PC2 and a part of the image PC3 overlap.

The intersection KS of the image PC exists in the overlapping area OB. The intersection KS1 existing in the overlapping region OB1 is the intersection KS between the image PC1 and the image PC2. The intersection KS2 existing in the overlapping region OB2 is the intersection KS between the image PC2 and the image PC3. The intersection extraction unit 413 extracts the intersection KS of a plurality of two-dimensional image PCs photographed by the image pickup apparatus 30.

FIG. 6 is a schematic diagram for explaining an example of processing of the measurement system 50 according to the present embodiment. FIG. 6 is a diagram showing an example of image PCs (PC1, PC2) taken while the swivel body 3 is swiveling. The intersection extraction unit 413 extracts the intersection KS from the image PC.

As shown in FIG. 6, the intersection extraction unit 413 is composed of the image PC1 of the construction target SB arranged in the photographing area FM1 and the image PC2 of the construction target SB arranged in the photographing area FM2, from the image PC1 and the image PC2. Extract the intersection KS1 with. The imaging position calculation unit 416 calculates the estimated angle θs (Δθ1 in FIG. 5) of the swivel body 3 based on the common portion KS1 extracted by the common portion extraction unit 413. Further, the imaging position calculation unit 416 calculates the estimated position Ps (PJ2 in FIG. 5) of the imaging device 30 at the time of shooting based on the estimated angle θs.

The intersection KS is a feature point in the image PC. The intersection extraction unit 413 extracts the intersection KS based on a known feature point detection algorithm such as ORB (Oriented FAST and Rotated BRIEF) or Harris corner detection. The intersection extraction unit 413 extracts a plurality of feature points from each of the plurality of image PCs, and extracts a common portion KS by searching for similar feature points from the extracted plurality of feature points. The intersection extraction unit 413 may extract a plurality of points of the intersection KS. The intersection extraction unit 413 extracts, for example, the corners of the construction target SB in the image PC as feature points.

The swivel of the swivel body 3 causes the imaging region FM of the image pickup apparatus 30 to move in the swivel direction RD, and the movement of the imaging region FM displaces the intersection KS in the image PC. In the example shown in FIG. 6, the intersection KS exists at the pixel position PX1 in the image PC1 and exists at the pixel position PX2 in the image PC2. The position where the intersection KS exists differs between the image PC1 and the image PC2. That is, the intersection KS1 is displaced between the image PC1 and the image PC2. The image pickup position calculation unit 416 can calculate the turning angle Δθ1 from the position PJ1 to the position PJ2 based on the position of the common portion KS in the plurality of image PCs.

FIG. 7 is a schematic diagram for explaining an example of processing of the measurement system 50 according to the present embodiment. FIG. 7 is a diagram illustrating that the image pickup device coordinate system (Xs, Ys, Zs) moves as the swivel body 3 swivels. As shown in FIG. 7, the swivel body 3 swivels around the swivel axis Zr in the Xm-Ym plane of the vehicle body coordinate system. When the swivel body 3 swivels about the swivel axis Zr by the swivel angle Δθ, the image pickup device coordinate system (Xs, Ys, Zs) moves about the swivel axis Zr by the swivel angle Δθ. The imaging position calculation unit 416 determines the estimated angle θs of the swivel body 3 and the imaging device 30 when the swivel body 3 swivels by the swivel angle Δθ from the pre-swirl angle θra on the condition that the swivel body 3 swivels around the swivel axis Zr. The estimated position Ps can be calculated.

[Calculation of estimated angle]
FIG. 8 is a schematic diagram for explaining an example of processing of the measurement system 50 according to the present embodiment. FIG. 8 is a schematic view showing an example of calculating the turning angle θ of the swivel body 3 and the position P of the imaging device 30. FIG. 8 shows the turning angle θ of the turning body 3 and the position P of the image pickup device 30 in the vehicle body coordinate system.

In the example shown in FIG. 8, the swivel body 3 swivels in the swivel direction RD in the order of the pre-turn angle θra, the first estimated angle θs1, the second estimated angle θs2, the third estimated angle θs3, and the post-turn angle θrb. The image pickup apparatus 30 moves in the order of the pre-swivel position Pra, the first estimated position Ps1, the second estimated position Ps2, the third estimated position Ps3, and the post-spin position Prb due to the swivel of the swivel body 3. The imaging region FM of the imaging device 30 moves in the order of the imaging region FMra, the imaging region FMs1, the imaging region FMs2, the imaging region FMs3, and the imaging region FMrb by moving the imaging device 30 in the turning direction RD.

When the angle before turning θra and the angle after turning θrb, the swivel body 3 is in the swivel stop state. The turning body 3 in the turning stopped state at the turning pre-turn angle θra reaches the turning post-turn angle θrb via the first estimated angle θs1, the second estimated angle θs2, and the third estimated angle θs3. It turns from θra to the angle θrb after turning.

The image pickup apparatus 30 acquires the image PCra of the construction target SB arranged in the photographing area FMra while being arranged at the position Pra before turning. The image pickup apparatus 30 acquires the image PCs1 of the construction target SB arranged in the photographing area FMs1 in the state of being arranged at the first estimated position Ps1. The image pickup apparatus 30 acquires the image PCs2 of the construction target SB arranged in the photographing area FMs2 while being arranged at the second estimated position Ps2. The image pickup apparatus 30 acquires the image PCs3 of the construction target SB arranged in the photographing area FMs3 while being arranged at the third estimated position Ps3. The image pickup apparatus 30 acquires the image PCrb of the construction target SB arranged in the photographing area FMrb while being arranged at the position Prb after turning.

As described above, the imaging position calculation unit 416 performs before turning based on the position and posture of the turning body 3 detected by the position detection device 23 and the posture detection device 24 in the turning stop state before the turning body 3 starts turning. Calculate the angle θra. Further, the imaging position calculation unit 416 determines the post-swing angle θrb based on the position and posture of the swivel body 3 detected by the position detection device 23 and the posture detection device 24 when the swivel body 3 is in the swivel stop state after the swivel is completed. calculate. Further, the imaging position calculation unit 416 calculates the pre-swivel position Pra based on the pre-swivel angle θra. Further, the imaging position calculation unit 416 calculates the post-swivel position Prb based on the post-swivel angle θrb.

The angle before turning θra, the angle after turning θrb, the position before turning Pra, and the position after turning Prb are calculated with high accuracy based on the detection data of the position detection device 23 and the detection data of the posture detection device 24.

As described above, the imaging position calculation unit 416 has an estimated angle θs (first estimated angle θs1, second estimated angle θs2, based on the intersection KS of the plurality of image PCs captured while the swivel body 3 is swirling. And the third estimated angle θs3) is calculated. Further, the imaging position calculation unit 416 calculates the estimated position Ps (first estimated position Ps1, second estimated position Ps2, and third estimated position Ps3) based on the estimated angle θs.

The intersection extraction unit 413 extracts the intersection KS1 between the image PCra and the image PCs1. The imaging position calculation unit 416 turns based on the pre-swing angle θra of the swivel body 3 and the intersection KS1 of the image PCra and the image PCs1 on the condition that the swivel body 3 swivels around the swivel axis Zr. Calculate the angle Δθ1. The first estimated angle θs1 is the sum of the correct answer angle before turning θra and the turning angle Δθ1 (θs1 = θra + Δθ1).

Similarly, the intersection extraction unit 413 can calculate the turning angle Δθ2, the turning angle Δθ3, and the turning angle Δθ4, and the imaging position calculation unit 416 has the second estimated angle θs2 (θs2 = θra + Δθ1 + Δθ2) and the third estimation. The angle θs3 (θs3 = θra + Δθ1 + Δθ2 + Δθ3) and the fourth estimated angle θs4 (θs4 = θra + Δθ1 + Δθ2 + Δθ3 + Δθ4) can be calculated.

In this way, the image pickup position calculation unit 416 can stepwise calculate the estimated angles θs (θs1, θs2, θs3, θs4) of the swivel body 3 at the time of shooting by the image pickup device 30 while the swivel body 3 is swiveling. .. Further, by calculating the estimated angle θs of the swivel body 3, the imaging position calculation unit 416 determines the estimated position Ps (Ps1, Ps2, Ps3) of the imaging device 30 while the swivel body 3 is swirling based on the estimated angle θs. , Ps4) can be calculated.

Further, the image pickup position calculation unit 416 acquires time point data indicating the time point at which each of the plurality of image PCs (PCra, PCs1, PCs2, PCs3, PCs4, PCrb) is photographed from the image pickup apparatus 30. The image pickup apparatus 416 can calculate the turning speed V based on the time when each of the plurality of image PCs is photographed and the turning angles Δθ (Δθ1, Δθ2, Δθ3, Δθ4). For example, the swivel body 3 is based on the time from the time when the image PCs1 is photographed to the time when the image PCs2 is photographed and the amount of movement from the estimated angle θs1 to the estimated angle θs2 when the image PCs1 is photographed. The turning angle V when turning from the estimated angle θs1 to the estimated angle θs2 can be calculated. Further, the image pickup apparatus 416 can calculate the turning direction R based on the time when each of the plurality of image PCs is photographed and the turning angle Δθ.

As described above, in the present embodiment, the image pickup position calculation unit 416 can calculate the turning angle θ based on the intersection KS of the plurality of image PCs.

[Correction of estimated angle]
The post-turn angle θrb is calculated with high accuracy based on the detection data of the position detection device 23 and the detection data of the posture detection device 24. On the other hand, the estimated angle θs4 is calculated based on the displacement amount of the intersection KS. If the estimated angle θs4 and the estimated position Ps4 calculated using the intersection KS are accurate, the difference between the estimated angle θs4 and the post-turn angle θrb is small, and the difference between the estimated position Ps4 and the post-turn position Prb is small.

On the other hand, for example, due to the cumulative error of the displacement amount of the intersection KS, the error of the estimated angle θs4 and the estimated position Ps4 calculated using the intersection KS may become large. If the error between the estimated angle θs4 and the estimated position Ps4 is large, the difference between the estimated angle θs4 and the post-turning angle θrb becomes large, and the difference between the estimated position Ps4 and the post-turning position Prb becomes large.

When there is a difference between the estimated angle θs4 and the angle after turning θrb, the imaging position calculation unit 416 is turning based on the angle θr (angle before turning θra and angle after turning θrb) which is the turning angle θ in the turning stopped state. The estimated angle θs (first estimated angle θs1, second estimated angle θs2, third estimated angle θs3, and fourth estimated angle θs4), which is the turning angle θ in the above, is corrected. Further, the imaging position calculation unit 416 corrects the estimated position Ps based on the corrected estimated angle θs.

The image pickup position calculation unit 416 can accurately calculate the estimated position Ps1 of the image pickup apparatus 30 at the time of photographing the image PCs1 based on the corrected estimated angle θs1'. Similarly, the imaging position calculation unit 416 can accurately calculate the estimated position Ps2 of the imaging device 30 at the time of photographing the image PCs2 based on the corrected estimated angle θs2', and the corrected estimated angle θs3' Based on the above, the estimated position Ps3 of the image pickup apparatus 30 at the time of photographing the image PCs3 can be calculated with high accuracy. Therefore, the three-dimensional position calculation unit 417 determines the construction target SB of the site coordinate system based on the estimated position Ps of the image pickup device 30 at the time of shooting and the image PCs captured by the image pickup device 30 calculated with high accuracy. The three-dimensional position can be calculated with high accuracy.

[Division of extraction target area and flattening of histogram]
Next, the division of the extraction target area WD will be described. In FIG. 6, the common part extraction unit 413 has described an example of extracting the common part KS from the entire image PC. The common part extraction unit 413 may set an extraction target area WD for extracting the common part KS in a part of the image PC, and extract the common part KS from the extraction target area WD, for example. Hereinafter, the case where the extraction target area WD is set in the central portion of the image PC will be described. The common portion extraction unit 413 can divide the extraction target region WD set in the central portion of the image PC into a plurality of divided regions, and can extract the common portion KS from each of the plurality of divided regions.

FIG. 9 is a diagram schematically showing a state in which one extraction target region WD is set in the image PC. FIG. 9 is an image PC imaged by, for example, a downward-facing image pickup apparatus 30C. When the feature points (intersection KS) are extracted by the feature point detection algorithm such as the ORB described above, in the extraction target region WD as shown in FIG. 9, depending on the shape of the construction target SB or the shooting conditions of the image PC. The intersection KS may be concentrated in a specific area without being dispersed. The reason why the intersection KS is concentrated in a specific region is that, depending on the feature point detection algorithm, the intersection KS is extracted in order from the point where the degree of feature (feature amount) is high. Therefore, when the image PC has a portion having high contrast, for example, when the construction target SB has an artificial straight portion, a corner portion, an arc portion, or the like, the common portion KS is concentrated in that portion. It ends up. For example, as shown in FIG. 9, when the construction target SB is excavated by the bucket of the hydraulic excavator 1 and the hole HL is formed in the construction target SB, the inside of the hole HL becomes a dark part and the periphery of the hole HL is bright in the image PC. Become a department. Since the edge of the hole HL, which is the boundary between the dark part and the bright part, is a portion having high contrast in the image PC, there is a high possibility that the common portion KS is extracted so as to concentrate on the edge of the hole HL. Further, since the contrast is low in the dark portion inside the hole HL, it is difficult to extract the common portion KS.

If the intersection KS, which is a feature point, is concentrated in a specific region of the image PC, the accuracy of the estimated angle θs and the estimated position Ps calculated based on the intersection KS may decrease. On the other hand, since the intersection KS, which is a feature point, is dispersed in the image PC or the extraction target region WD, the accuracy of the estimated angle θs and the estimated position Ps calculated based on the intersection KS is improved.

Therefore, the common portion extraction unit 413 divides the extraction target region WD of the image PC into a plurality of divided regions, and extracts the common portion KS from each of the plurality of divided regions. As a result, the intersection KS extracted in the image PC is dispersed. For example, when 100 points of intersection KS are extracted in order from the point with the highest feature amount based on the feature point detection algorithm, if the extraction target area WD is not divided, 100 points are set at the edge of the hole HL as shown in FIG. The common part KS of is concentrated. On the other hand, when the extraction target region WD is divided into, for example, 25 points of common portion KS are extracted in order from the point having the highest feature amount in each of the divided regions divided into four. Therefore, the intersection KS extracted in the image PC is dispersed. The imaging position calculation unit 416 can calculate the estimated angle θs and the estimated position Ps with high accuracy based on the dispersed intersection KS.

FIG. 10 is a diagram showing an example of the extraction target region WD of the image PC according to the present embodiment. As shown in FIG. 10, the common partial extraction unit 413 divides the extraction target region WD of the image PC into a plurality of division regions WD1, WD2, WD3, WD4, and divides the plurality of division regions WD1, WD2, WD3, WD4. From each, the intersection KS of a plurality of image PCs is extracted. The imaging position calculation unit 416 acquires the estimated angle θs and the estimated position Ps at the time of shooting based on the common portions KS1, KS2, KS3, KS4 extracted from each of the plurality of divided regions WD1, WD2, WD3, WD4. ..

The intersection extraction unit 413 extracts at least one intersection KS from each of the plurality of divided regions WD1, WD2, WD3, and WD4. The intersection extraction unit 413 extracts a predetermined number or more of the intersection KS from each of the plurality of divided regions WD1, WD2, WD3, WD4. FIG. 10 shows an example in which at least six common intersections KS (KS1, KS2, KS3, KS4) are extracted from each of the plurality of divided regions WD1, WD2, WD3, WD4. As a result, the intersection KS extracted in the image PC is dispersed.

In the present embodiment, the extraction target area WD has a rectangular shape. The center of the extraction target region WD in the turning direction RD is divided by the first dividing line, and the center of the extraction target region WD in the vertical direction orthogonal to the turning direction RD is divided by the second dividing line. The outer shape and area of the divided regions WD1, WD2, WD3, and WD4 are the same.

The shapes of the extraction target region WD and the division regions WD1, WD2, WD3, and WD4 can be arbitrarily determined. The extraction target area WD does not have to be rectangular, may be square, or may be circular. Further, the outer shapes and areas of the divided regions WD1, WD2, WD3, and WD4 may be the same or different. Further, the number of divided regions is not limited to four, and can be set to any plurality.

As described above, since the contrast is low in the dark part such as the inside of the hole HL, it is difficult to extract the intersection KS. Therefore, when the extraction target area WD is divided into a plurality of divided areas, if a portion having a low contrast among the construction target SBs is arranged in the divided area, it is difficult to extract the common portion KS in the dark part in the divided area. Become.

Therefore, the image processing unit 412 corrects the contrast distribution of the image PC. The image processing unit 412 corrects the contrast distribution of the image PC based on, for example, a histogram flattening algorithm. The intersection extraction unit 413 extracts the intersection KS of a plurality of image PCs from the extraction target area WD of the image PC whose contrast distribution has been corrected. As a result, even if a portion having a low contrast exists in the divided region, the contrast distribution is corrected so that the contrast of the divided region becomes high. By increasing the contrast of the divided region, the common portion extraction unit 413 can extract the common portion KS from the divided region.

FIG. 11 is a diagram in which the intersection KS is extracted in the image PC in which the contrast distribution of each divided region is corrected by the histogram flattening algorithm. Histogram flattening refers to a process of converting an image PC so that a histogram showing the relationship between the gradation and frequency of each pixel of the image PC is evenly distributed over the entire gradation.

In FIG. 11, the image processing unit 412 corrects the brightness of each pixel in the divided region to improve the contrast of the image PC, so that the common partial extraction unit 412 has holes in the divided regions WD1, WD2, WD3, WD4. The intersection KS can be extracted in HL.

As shown in FIG. 9, the extraction target area WD is defined in the central portion of the image PC. In the image PC, a frame-shaped non-target area WE is defined around the extraction target area WD.

Since the extraction target area WD is defined in the central portion of the image PC in the turning direction RD, when the swivel body 3 turns right or left, the common portion KS extracted in the extraction target area WD is the extraction target. Move to the non-target area WE on the left or right side of the area WD. That is, when the swivel body 3 is swiveled, the common portion KS extracted in the extraction target region WD does not immediately disappear from the image PC but moves to the non-target region WE, so that the common portion KS is continuously extracted. It is possible to estimate the turning angle θ with high accuracy.

Further, the extraction target area WD is set in the vertical direction so that the traveling body 5 is located outside the extraction target area WD in the image PC, in other words, the image of the traveling body 5 does not enter the extraction target area WD. It is defined in the central part of the image PC. As a result, the extraction of the feature points of the traveling body 5 is suppressed, and the intersection extraction unit 413 can extract more feature points of the construction target SB.

The image processing unit 412 may correct the contrast distribution for each of the plurality of divided regions WD1, WD2, WD3, WD4, or may correct the contrast distribution of the entire undivided extraction target region WD. , The contrast distribution may be corrected in the image PC.

The extraction target area WD may be set at a position deviated from the central portion of the image PC. Further, the entire area of the image PC may be set as the extraction target area WD without providing the non-target area WE.

[Input device]
In the present embodiment, the input device 32 has a swivel continuous shooting switch 32A capable of generating a shooting start command signal for commanding the start of the swivel continuous shooting mode and a shooting end command signal for commanding the end of the swivel continuous shooting mode.

[Measurement method]
FIG. 12 is a flowchart showing an example of the measurement method according to the present embodiment. FIG. 13 is a timing chart of the measurement method according to the present embodiment.

The driver of the hydraulic excavator 1 operates the operation device 35 to turn the swivel body 3 so that the image pickup device 30 faces the measurement start position of the construction target SB. When a predetermined time elapses from the time point t0 when the operation of the hydraulic excavator 1 including the turning of the turning body 3 is stopped and the time point t1 is reached, the determination unit 418 moves the position detection device 23 and the posture detection device 24, respectively. Is determined to be in a statically indeterminate state in which the detected data can be output stably.

When the hydraulic excavator 1 is in the stopped operation state and the position detection device 32 and the posture detection device 24 are in the static state and the imaging start command signal is acquired, the imaging position calculation unit 416 moves the position of the swivel body 3 from the position detection device 23. The detection data indicating the posture of the swivel body 3 is acquired from the posture detection device 24 (step S10).

The imaging position calculation unit 416 acquires the angle before turning θra and the position Pra before turning.

The detection data of the position detection device 23 and the detection data of the posture detection device 24 are temporarily stored in the storage device 42.

When starting the turning continuous shooting mode, the driver of the hydraulic excavator 1 operates (presses) the turning continuous shooting switch 32A. In the example shown in FIG. 13, the swivel continuous photographing switch 32A is operated at the time point t2. The shooting start command signal generated by operating the swivel continuous shooting switch 32A is output to the arithmetic processing unit 41. The signal acquisition unit 411 acquires a shooting start command signal (step S20).

Further, when the swivel continuous shooting switch 32A is operated and a shooting start command signal is acquired, the arithmetic processing unit 41 starts imaging of the imaging device 30 (step S30). The image acquisition unit 410 acquires the image PCra of the target SB taken by the image pickup apparatus 30 in the operation stopped state before the hydraulic excavator 1 starts the operation. In addition, the image storage unit 423 stores the image PCra.

The driver operates the operating device 35 to start turning the turning body 3 in a state where the running body 5 has stopped running (step S40). The driver operates the operation device 35 so that the image pickup device 30 turns from the turn start position facing the measurement start position of the target SB to the turn end position where the image pickup device 30 steps on the measurement end position of the target SB. Then, the turning of the swivel body 3 is started.

In the example shown in FIG. 13, the operating device 35 is operated at the time point t3, and the turning body 3 starts turning. Each of the plurality of image pickup devices 30 (30A, 30B, 30C, 30D) captures the image PCs of the target SB a plurality of times at intervals of time while the swivel body 3 is in the swivel state.

The image acquisition unit 410 sequentially acquires a plurality of image PCs of the construction target SB photographed by the image pickup apparatus 30 while the rotating body 3 is rotating. Further, the image storage unit 423 sequentially stores a plurality of image PCs.

When the turning body 3 reaches the turning end position, the driver releases the operation of the operating device 35 and ends the turning of the turning body 3 (step S60). In the example shown in FIG. 13, the operation of the operating device 35 is released at the time point t4, and the turning of the swivel body 3 is completed.

When a predetermined time elapses from the time point t4 when the turning of the turning body 3 is stopped and the time point t5 is reached, the determination unit 418 stabilizes the detection data by each of the position detection device 23 and the posture detection device 24. It is determined that the statically indeterminate state can be output.

When the hydraulic excavator 1 is in the stopped operation state and the position detection device 32 and the posture detection device 24 are in the static state, the photographing position calculation unit 416 acquires detection data indicating the position of the swivel body 3 from the position detection device 23. The detection data indicating the posture of the swivel body 3 is acquired from the posture detection device 24 (step S70).

The imaging position calculation unit 416 acquires the angle after turning θrb and the position after turning Prb.

The detection data of the position detection device 23 and the detection data of the posture detection device 24 are temporarily stored in the storage device 42.

Further, the image acquisition unit 410 acquires the image PCrb of the construction target SB taken by the image pickup apparatus 30 in the operation stopped state after the operation of the hydraulic excavator 1 is completed. Further, the image storage unit 423 stores the image PCrb.

When ending the turning continuous shooting mode, the driver of the hydraulic excavator 1 operates (presses) the turning continuous shooting switch 32A. In the example shown in FIG. 13, the swivel continuous photographing switch 32A is operated at the time point t6. The shooting end command signal generated by operating the swivel continuous shooting switch 32A is output to the arithmetic processing unit 41. The signal acquisition unit 411 acquires a shooting end command signal (step S80).

When the shooting end signal is acquired, the shooting of the imaging device 30 is completed (step S90). The image pickup apparatus 30 transitions to a state in which photography is not possible.

The intersection extraction unit 413 extracts the intersection KS from each of the plurality of image PCs stored in the image storage unit 423 (step S120).

The intersection extraction unit 413 extracts the intersection KS of the two image PCs from at least two image PCs taken by the image pickup apparatus 30 while the swivel body 3 is stopped and swiveled. The extraction of the common portion KS may be performed for all the image PCs acquired from the start to the end of the turning of the turning body 3 in the turning direction R of the turning body 3, or based on a predetermined rule. It may be carried out for the image PC selected in the above.

As described with reference to FIG. 10, the common partial extraction unit 413 divides the extraction target area WD of the image PC into a plurality of divided areas WD1, WD2, WD3, WD4, and divides the plurality of divided areas WD1, WD2, A specified number or more of common part KSs are extracted from each of WD3 and WD4.

Further, as described with reference to FIG. 11, the intersection extraction unit 413 can correct the contrast distribution of the image PC by using the histogram flattening algorithm. The intersection extraction unit 413 can extract the intersection KS from the extraction target region WD of the image PC whose contrast distribution has been corrected.

Next, the imaging position calculation unit 416 estimates the estimated angle θs of the swivel body 3 based on the intersection KS of the plurality of image PCs (step S140). The imaging position calculation unit 416 can calculate the turning angle Δθ based on the position of the common portion KS in the plurality of image PCs to estimate the estimated angle θs.

When the difference between the estimated angle θs after the end of turning and the angle θrb after turning is less than the specified value, the imaging position calculation unit 416 is in the process of turning based on the angle θr (angle before turning θra, angle after turning θrb). Correct the estimated angle θs. Further, the imaging position calculation unit 416 corrects the estimated position Ps of the imaging device 30 during rotation based on the corrected estimated angle θs. The imaging position calculation unit 416 can correct the estimated angle θs based on the procedure described above.

The three-dimensional position calculation unit 417 stereo-processes the image PC at the time of shooting to calculate the three-dimensional position in the image pickup device coordinate system of the plurality of construction target SBs (step S160). Further, the three-dimensional position calculation unit 417 converts the three-dimensional position in the image pickup apparatus coordinate system into the three-dimensional position in the field coordinate system.

[effect]
As described above, according to the present embodiment, when the position of the photographing apparatus 30 is acquired based on the common portion KS, which is a feature point common to a plurality of image PCs, the extraction target region WD for extracting the common portion KS. Is divided into a plurality of division areas WD1, WD2, WD3, WD4. The intersection extraction unit 413 extracts the intersection KS from each of the plurality of divided regions WD1, WD2, WD3, and WD4. As a result, the intersection KS extracted in the image PC is dispersed. Therefore, the image pickup position calculation unit 416 can calculate the estimated position Ps of the image pickup apparatus 30 with high accuracy based on the distributed intersection KS. Therefore, a decrease in measurement accuracy of the three-dimensional shape of the target SB is suppressed.

Further, in the present embodiment, the contrast distribution of the image PC is corrected by the image processing unit 412. As a result, even if a portion having a low contrast exists in the extraction target region WD, the contrast distribution is corrected so that the contrast of the extraction target region WD becomes high. By increasing the contrast of the extraction target region WD, the common portion extraction unit 413 can extract the common portion KS from the extraction target region WD. Therefore, the image pickup position calculation unit 416 can calculate the estimated position Ps of the image pickup apparatus 30 with high accuracy based on the extracted common portion KS.

Second embodiment.
The second embodiment will be described. In the following description, the same or equivalent components as those in the above-described embodiment are designated by the same reference numerals, and the description thereof will be simplified or omitted.

FIG. 14 is a schematic diagram for explaining an example of processing of the measurement system 30 according to the present embodiment. In the above-described embodiment, the fact that the hydraulic excavator 1 is in operation means that the traveling body 5 is stopped and the swivel body 3 is turning. The fact that the hydraulic excavator 1 is operating may mean that the swivel body 3 is turning and the traveling body 5 is running, or that the swivel body 3 is turning and the traveling body 5 is running. It may be.

As shown in FIG. 14, when the traveling body 5 travels in the traveling direction MD, the imaging device 30 and the photographing region FM of the imaging device 30 move in the traveling direction MD. The photographing device 30 photographs the target SB so that an overlapping area is provided in the photographing area FM1 and the photographing area FM2, and the overlapping area is provided in the photographing area FM1 and the photographing area FM2. The photographing device 30 photographs the images PC1, PC2, and PC3 of the target SB arranged in each of the photographing areas FM1, FM2, and FM3.

The intersection extraction unit 413 can extract the intersection KS1 between the image PC1 and the image PC2 and the intersection KS2 between the image PC2 and the image PC3.

The image pickup position calculation unit 416 calculates the movement amount ΔD1 of the image pickup apparatus 30 based on the displacement amount of the common portion KS1. Further, the image pickup position calculation unit 416 calculates the movement amount ΔD2 of the image pickup apparatus 30 based on the displacement amount of the common portion KS2. The image pickup position calculation unit 416 acquires the position of the image pickup device 30 at the time of shooting while the traveling body 5 is traveling, based on the movement amount Δ (ΔD1, ΔD2) of the image pickup device 30. In the above-described embodiment, the image pickup position calculation unit 416 calculated the turning angle θ, but in the present embodiment, the position of the image pickup device 30 is the position in the X-axis direction, the position in the Y-axis direction, and the Z-axis direction. 6 variables of position, roll angle, pitch angle, and yaw angle are calculated. The three-dimensional position calculation unit 417 calculates the three-dimensional position of the target SB based on the position of the image pickup device 30 at the time of shooting and the image PC taken by the image pickup device 30.

In the above-described embodiment, the position of the image pickup apparatus 30 is calculated based on the common portion KS while the hydraulic excavator 1 is turning, but the present invention is not limited to that embodiment. For example, the position of the image pickup device 30 while the hydraulic excavator 1 is turning may be calculated based on the detection data of the position detection device 23 and the detection data of the posture detection device 24.

In the above-described embodiment, the imaging position calculation unit 416 calculates the imaging position with the swivel axis Zr as a constraint condition and the variable as only the swivel angle θ, but the imaging device does not use the swivel axis Zr as a restraint condition. The position and orientation of 30 may be calculated based on six variables of the position in the X-axis direction, the position in the Y-axis direction, the position in the Z-axis direction, the roll angle, the pitch angle, and the yaw angle.

In the above-described embodiment, the input device 32 may be attached to at least one of the right operating lever 35R and the left operating lever 35L of the operating device 35, or is provided on the monitor panel arranged in the driver's cab 4. It may be provided, or it may be provided in the portable terminal device. Further, the input device 32 may be provided outside the hydraulic excavator 1 and the start or end of the photographing of the image pickup device 30 may be remotely controlled.

In the above-described embodiment, the imaging position calculation unit 416 calculates the turning angle based on the intersection KS, but the imaging position calculation unit 416 may calculate the turning angle based on the detection result of the detection device. Good. For example, the imaging position calculation unit 416 may calculate the turning angle θ of the swivel body 3 based on the detection data of the position detection device 23 or the detection data of the posture detection device 24, or the detection data of the operation amount sensor 36. The turning angle θ may be calculated based on the above, or the turning angle θ may be calculated based on the detection data of an angle sensor capable of detecting the turning angle of the turning body 3, for example, a rotary encoder. In this case, the arithmetic processing unit 41 synchronizes the timing of acquiring the turning angle θ, which is the detection value of the angle detection sensor, with the timing of at least a pair of imaging devices 30 photographing the construction target SB. In this way, the timing at which the image PC is captured by at least a pair of image pickup devices 30 is associated with the turning angle θ of the swivel body 3 at that timing.

In the above-described embodiment, the arithmetic processing unit 41 stereo-processes the image PCs imaged by at least a pair of image pickup devices 30 to realize three-dimensional measurement, but the present invention is not limited to such an embodiment. For example, the image PC of the excavator SB around the hydraulic excavator 1 imaged by at least a pair of image pickup devices 30, and the position and posture of the hydraulic excavator 1 obtained by the position detection device 23 and the posture detection device 24 at rest. Is transmitted to, for example, an external management device (portable terminal, server device, etc.) of the hydraulic excavator 1. Then, an external management device stereo-processes the image PC of the excavation target SB around the hydraulic excavator 1, and obtains the turning angle θ at the time of turning the swivel body 3 and the position and posture of the hydraulic excavator 1, and the obtained results are obtained. May be used to obtain the three-dimensional position of the excavation target SB around the hydraulic excavator 1 during turning. In this case, the external management device of the hydraulic excavator 1 corresponds to the arithmetic processing unit 41. That is, in another embodiment, the measurement system 50 does not have to realize all the functions only in the hydraulic excavator 1, and even if the external management device has some functions, for example, the functions of the arithmetic processing unit 41. Good.

In the above embodiment, the image pickup device is a stereo camera including at least a pair of image pickup devices 30. When shooting with a stereo camera, the shooting timing of each camera is synchronized. The image pickup apparatus may be capable of stereo shooting with one camera. That is, it may be an imaging device capable of stereo processing based on two images having different shooting timings by one camera. The imaging device is not limited to a stereo camera. The image pickup apparatus may be a sensor that can obtain both an image and three-dimensional data, such as a TOF (Time Of Flight) camera. The image pickup device may be an image pickup device in which three-dimensional data can be obtained by one camera. The image pickup device may be an image pickup device capable of stereo measurement by one camera. The imaging device may be a laser scanner.

In the above-described embodiment, the work machine 1 is a hydraulic excavator 1 having a swivel body 3. The work machine 1 may be a work machine having no swivel body. For example, the work machine may be at least one of a bulldozer, a wheel loader, a dump truck, and a motor grader.

1 ... hydraulic excavator (working machine), 1B ... car body, 2 ... working machine, 3 ... swivel body, 4 ... driver's cab, 4S ... driver's seat, 5 ... running body, 5A, 5B ... Arm, 8 ... bucket, 9 ... counter weight, 10 ... boom cylinder, 11 ... arm cylinder, 12 ... bucket cylinder, 21 ... GPS antenna, 23 ... position detection device, 24 ... attitude detection device, 30 ... image pickup device, 30A, 30B, 30C, 30D ... Imaging device, 31 ... Hub, 32 ... Input device, 32A ... Swivel continuous shooting switch, 35 ... Operating device, 35L ... Left operation lever, 35R ... Right operation lever, 36 ... Operation amount sensor, 37 ... Swivel sensor, 40 ... control device, 41 ... arithmetic processing device, 42 ... storage device, 43 ... input / output interface, 50 ... measurement system, 300 ... stereo camera, 301 ... first stereo camera, 302 ... second stereo camera, 410 ... image acquisition unit, 411 ... signal acquisition unit, 412 ... image processing unit, 413 ... common part extraction unit, 416 ... imaging position calculation unit, 417 ... three-dimensional position calculation unit, 418 ... judgment unit, 423 ... image storage unit, FM ... Shooting area, KS ... Common part, MD ... Traveling direction, PC ... Image, PK ... Calculation image, Pr ... Turning position, Pra ... Before turning position, Prb ... After turning position, Ps ... Estimated position, RD ... Turning Direction, SB ... Construction target, WD ... Extraction target area, WD1, WD2, WD3, WD4 ... Divided area, WE ... Non-target area, Zr ... Turning axis, θr ... Turning angle, θra ... Before turning angle, θrb ... After turning Angle, θs ... Estimated angle.

Claims (3)

An image acquisition unit that acquires multiple images of the construction target taken by an image pickup device mounted on the rotating body while the work machine is rotating, and an image acquisition unit.
An extraction target area for extracting common parts in the plurality of images is divided into divided areas, and a common part extraction unit for extracting the common parts from each of the divided areas.
Based on the position data of the work machine detected by the position detection unit that detects the position of the work machine, the posture data of the work machine detected by the posture detection unit that detects the posture of the work machine, and the common part. Then, an imaging position calculation unit that calculates the position and orientation of the imaging device at the time when the image is taken, and
A three-dimensional position calculation unit that calculates the three-dimensional position of the construction target based on the position data, the posture data, and the image.
An image processing unit for correcting the contrast distribution of the divided region of the image is provided.
The common portion extraction unit extracts the common portion from the image in which the contrast distribution of the divided region is corrected.
Work machine measurement system. The work machine has a traveling body and
The extraction target area is defined in the image so that the traveling body is located outside the extraction target area.
The measurement system for a work machine according to claim 1. A work machine including the measurement system for the work machine according to claim 1 or 2.

Images