JP6433664B2 - Overhead image display device for construction machinery - Google Patents

Description

  The present invention relates to an overhead image display device mounted on a construction machine.

In an emergency disaster recovery work, when a remotely operated construction machine is operated remotely, the operator needs to operate it from a safe place sufficiently away from the work place.
Although it is possible to operate visually in places where the construction range is narrow, it is usually possible to remotely display an image from a video camera mounted on a construction machine, an image from a moving camera car, an image from a fixed camera, etc. Construction can be done by operating.
However, these cameras are likely to cause blind spots every time the construction machine moves or turns, which is disadvantageous for remote operation.
On the other hand, driving support that provides a driver with a plurality of cameras that capture the periphery of a passenger car, processes the images supplied from each camera, creates an overhead image as seen from above the passenger car An apparatus is provided (see Patent Document 1).
Therefore, it is conceivable to apply an overhead image display device that creates and displays such an overhead image to a construction machine.
In this case, the plurality of cameras are attached to appropriate portions of the construction machine.

JP 2006-121587 A

Construction machinery travels on the ground by rotating and driving crawlers and tires.
Therefore, if the ground is flat, the construction machine and each camera attached to the construction machine are inclined at the same inclination angle as the inclination angle of the ground, and each camera takes an image of the surroundings of the construction machine in an inclined state. . Thereby, the bird's-eye view image display device can appropriately generate and display the bird's-eye view image viewed from above the construction machine.
On the other hand, when there is local unevenness on the ground, the construction machine temporarily tilts at an inclination angle different from the inclination angle of the ground when passing through the local unevenness portion, and then passes through the uneven portion. When finished, it returns to the inclination angle of the ground. Here, the local unevenness in the ground refers to, for example, a groove or a hole that is large enough to fit a crawler or a tire, or rubble or rock that a crawler or a tire rides on.
Therefore, when the construction machine passes through the local unevenness of the ground, each camera also temporarily tilts at an inclination angle different from the inclination angle of the ground, and then returns to the inclination angle of the ground. Will fluctuate temporarily.
Such a change in the imaging range of each camera is disadvantageous in causing a temporal disturbance in the overhead image generated by the overhead image display device and displaying an accurate overhead image.
In particular, since construction machines are larger than passenger cars, the overhead image range that covers the entire circumference of the construction machine is extremely wide. The impact on this will be significant.
In addition, since construction machines run on rough ground with rough terrain and rubble, the construction machines are greatly inclined, and the amount of change in the imaging range of each camera becomes large, and the imaging of each camera The influence of the range variation on the accuracy of the overhead view image becomes even greater.
The present invention has been made in view of such circumstances, and an object of the present invention is to provide a construction machine overhead image display apparatus that is advantageous in displaying an accurate overhead image without being affected by the inclination of the construction machine. There is.

To achieve the above object, the present invention provides a plurality of cameras mounted on a construction machine that is operated unattended by remote control , and the construction machine based on captured images around the construction machine obtained from the plurality of cameras. A bird's-eye view image display device for a construction machine, comprising: a bird's-eye view image generation unit that generates a bird's-eye view image of the entire circumference of the construction machine, and a display unit that displays the bird's-eye view image generated by the bird's-eye view image generation unit; The plurality of cameras include a front camera that images the front of the construction machine, a rear camera that images the rear of the construction machine, a left camera that images the left side of the construction machine, and a right side of the construction machine. And a right camera that captures the image of the construction machine so that an image covering the entire periphery including the front, rear, right, and left of the construction machine can be obtained without blind spots. La, the left camera, the right side camera is attached to the construction machine so that the optical axes of the cameras is adjustable around a vertical axis and horizontal axis, said construction machine ground generated during traveling An inclination angle detection means for detecting an inclination angle θ that is inclined with respect to the ground, and an overhead image in a state in which the construction machine is in a posture as the inclination of the ground on the ground is obtained based on the inclination angle θ. And an overhead image correction unit that corrects the overhead image as described above, and the inclination angle detection unit applies a captured image around the construction machine obtained in time series from at least two of the plurality of cameras. A feature point movement amount detecting means for extracting a feature point included, and detecting a movement amount of a three-dimensional position of the feature point based on a correspondence relationship between the feature points extracted in time series; and movement of the feature point Detection angle calculation means for calculating the inclination angle θ based on the angle, and the angle around the vertical axis and the angle around the horizontal axis of the optical axis of the camera are set based on the ground on which the construction machine is installed The image obtained by capturing the captured object with the camera is adjusted so as to match the reference line displayed on the screen, and the parameters used for the overhead image generation processing in the overhead image generation means are as follows: The object is adjusted based on an image obtained by imaging the object with the camera.

  According to the present invention, even if the construction machine tilts with respect to the ground by passing through the local unevenness of the ground, and the imaging range of each camera changes temporarily, the tilt of the construction machine This is advantageous for displaying an accurate overhead image without being affected by the angle θ.

It is a side view of a backhoe to which the overhead image display device for construction machines of an embodiment is applied. It is a top view of a backhoe. It is a block diagram which shows the structure of the remote control system which carries out unmanned operation of a backhoe by remote control. It is a figure which shows an example of a bird's-eye view image. It is a block diagram which shows the structure of the several camera and control part of a construction machine side unit. It is a functional block diagram of the control part of a construction machine side unit. It is an operation | movement flowchart of the overhead image display apparatus for construction machines.

(Embodiment)
Hereinafter, the present embodiment will be described with reference to the drawings.
First, a construction machine on which the overhead image display device for construction machine according to the present invention is mounted will be described.
In the present embodiment, a case will be described in which a construction machine used in a dangerous construction site such as an emergency disaster recovery construction is a backhoe that is operated unattended by remote control.
In this case, the operator remotely operates the backhoe from a safe place sufficiently away from the construction place.

First, the configuration of the backhoe will be described.
As shown in FIGS. 1 and 2, the backhoe 10 includes a lower traveling body 12, an upper swing body 14, a boom 16, an arm 18, and a bucket 20.

The lower traveling body 12 travels on the ground G by the rotation of the crawler 1202.
The upper turning body 14 is provided on the upper part of the lower traveling body 12 so as to be horizontally turnable around a turning axis.
The upper swing body 14 is provided with an operation chamber 1402, and the operation chamber 1402 travels the lower traveling body 12, swings the upper swing body 14, swings the boom 16, swings the arm 18, swings the bucket 20. A plurality of operation devices such as operation levers and operation pedals (not shown) are provided for operating the devices.

The boom 16 is swingably supported by the upper swing body 14 via a support shaft whose base end extends in the horizontal direction.
The arm 18 is swingably supported at the distal end of the boom 16 via a support shaft whose base end extends in the horizontal direction.
The bucket 20 is swingably supported at the distal end of the arm 18 via a support shaft whose base end extends in the horizontal direction.
A boom cylinder 1602 that swings the boom 16 is provided between the upper swing body 14 and the boom 16.
An arm cylinder 1802 that swings the arm 18 is provided between the boom 16 and the arm 18.
A bucket cylinder 2002 that swings the bucket 20 is provided between the arm 18 and the bucket 20.
These boom cylinder 1602, arm cylinder 1802, and bucket cylinder 2002 are hydraulic cylinders.
Therefore, the boom 16 is swung with respect to the upper swing body 14 by the expansion and contraction of the boom cylinder 1602.
Further, the arm 18 is swung with respect to the boom 16 by the expansion and contraction of the arm cylinder 1802.
Further, the bucket 20 is swung with respect to the arm 18 by the expansion and contraction of the bucket cylinder 2002.

Next, a remote control system for remotely controlling the backhoe 10 will be described.
FIG. 3 is a block diagram showing the configuration of the remote control system.
The remote control system 30 includes a remote operation unit 32 and a construction machine side unit 34.
The remote control unit 32 is provided at a location away from the work site where the backhoe 10 performs work.
The remote operation unit 32 includes an operation unit 36, a display unit 38, a control unit 40, and a communication unit 42.
The operation unit 36 is operated by a worker to perform remote control of the construction machine, and includes, for example, a joystick or an operation switch.
The display unit 38 displays a bird's-eye view image as shown in FIG. 4 transmitted from the construction machine side unit 34, and includes a display such as a liquid crystal display device.
In the present embodiment, the display unit 38 constitutes a display unit.
The control unit 40 generates an operation command corresponding to the operation performed on the operation unit 36 and supplies the operation command to the communication unit 42, and accepts an overhead image transmitted from the construction machine side unit 34 via the communication unit 42 and displays the display unit 38. To supply.
The communication unit 42 communicates an operation command and an overhead image with the communication unit 50 of the construction machine side unit 34.
These two communication units 42 and 50 are connected by a wireless line.
As the wireless line, various conventionally known wireless lines such as SS wireless can be used, and the type, configuration, and system of the wireless line are arbitrary. Further, a plurality of types of wireless lines may be combined, or one or more relay stations may be provided.

The construction machine side unit 34 is provided on the backhoe 10 and includes a drive unit 44, a plurality of cameras 46, a control unit 48, and a communication unit 50.
The drive unit 44 drives each operation device of the backhoe 10, and various conventionally known actuators such as an air cylinder and a motor can be used.
The plurality of cameras 46 includes a front camera 4602, a rear camera 4604, a left camera 4606, and a right camera 4608.

As shown in FIGS. 1 and 2, the front camera 4602 is mounted on the front part of the upper swing body 14 and images the front of the backhoe 10.
The rear camera 4604 is mounted on the rear part of the upper swing body 14 and images the rear of the backhoe 10.
The left camera 4606 is mounted on the left side of the upper swing body 14 and images the left side of the backhoe 10.
The right camera 4608 is mounted on the right side of the upper swing body 14 and images the right side of the backhoe 10.
In the present embodiment, the angle of view of each camera 46 is 180 degrees.
Note that the angle of view of each camera 46 may be less than 180 degrees as long as a bird's-eye view image described later can be generated. However, by setting the angle of view of each camera 46 to 180 degrees and installing the cameras 46 on the front, rear, left, and right sides of the upper swing body 14 as described above, an image covering the entire periphery of the upper swing body 14 (backhoe 10) is obtained. It can be obtained efficiently and without generating blind spots, which is advantageous in accurately generating a bird's-eye view image as shown in FIG.
Each camera 46 is attached to the upper swing body 14 via a pan head (not shown) so that the optical axis of each camera 46 can be adjusted around the vertical axis and the horizontal axis.

As shown in FIG. 5, the control unit 48 is configured by a computer.
The computer includes a CPU 4802, a ROM 4804, a RAM 4806, a hard disk device 4808, a keyboard 4810, a mouse 4812, a display 4814, an input / output interface 4816, and the like connected via an interface circuit (not shown) and a bus line 4801.
The ROM 4804 stores a predetermined control program and the like, and the RAM 4806 provides a working area.
The hard disk device 4808 stores a program for realizing an overhead image generation unit 52, an inclination angle detection unit 54, and an overhead image correction unit 56, which will be described later.
A keyboard 4810 and a mouse 4812 receive operation inputs from the operator.
The display 4814 displays an image, and is composed of, for example, a liquid crystal display device.
The input / output interface 4816 exchanges data with peripheral devices. In this embodiment, the input / output interface 4816 accepts captured images from the cameras 46.
The control unit 48 receives the operation command transmitted from the remote operation unit 32 via the communication unit 50, and operates the operation devices of the backhoe 10 by controlling the drive unit 44 based on the operation command. Is.
Further, the control unit 48 realizes an overhead view image generation means 52, an inclination angle detection means 54, and an overhead view image correction means 56 as shown in FIG. 6 by the CPU 4802 executing the program.

The bird's-eye view image generation unit 52 generates a bird's-eye view image (FIG. 4) around the backhoe 10 based on the captured images around the backhoe 10 supplied from each camera 46, and communicates these captured images. The data is transmitted to the communication unit 42 of the remote operation unit 32 via the unit 50.
The overhead image generation by the overhead image generation means 52 is performed by combining viewpoint conversion images obtained by performing viewpoint conversion processing on captured images obtained from the cameras 46. Such viewpoint conversion processing and viewpoint conversion images are performed. As the combination processing, various conventionally known methods can be employed.

The inclination angle detection means 54 detects an inclination angle θ at which the backhoe 10 generated during traveling is inclined with respect to the ground G.
In the present embodiment, the tilt angle detection means 54 includes a feature point movement amount detection means 54A and a detection angle calculation means 54B.
The feature point moving amount detection means 54A extracts feature points included in the captured image around the backhoe 10 obtained in time series from at least two cameras 56 out of the plurality of cameras 56, and the features extracted in time series. The amount of movement of the three-dimensional position of the feature point is detected based on the point correspondence.
As a method for detecting the movement amount of the three-dimensional position of such a feature point, for example, a conventionally known image processing method called SFM (Structure From Motion) can be used.
The detection angle calculation unit 54B calculates the tilt angle θ based on the movement amount of the feature point detected by the feature point movement amount detection unit 54A.

The bird's-eye view image correction means 56 is so viewed as to obtain an overhead view image in a state where the backhoe 10 is in the posture of the ground G on the ground G based on the inclination angle θ detected by the inclination angle detection means 54. The overhead image generated by the image generation means 52 is corrected.
The overhead image correction unit 56 determines whether or not the absolute value of the change amount per unit time of the inclination angle θ exceeds a predetermined threshold value, and when the absolute value of the change amount exceeds the threshold value, the overhead image correction unit 56 While correcting the image, the overhead image correction is canceled when the absolute value of the amount of change is equal to or less than a predetermined threshold value.

  In the present embodiment, the display unit 48, the control unit 40, and the communication unit 42 of the remote operation unit 32, and the overhead view image display for the construction machine are displayed by the communication unit 50, the control unit 48, and each camera 46 of the construction machine side unit 34. The device is configured.

Next, the operation of the overhead image display device for construction machines will be described with reference to the flowchart of FIG.
It is assumed that the angles around the vertical axis and the horizontal axis of the optical axis of each camera 46 have been calibrated in advance so that an overhead image can be obtained accurately.
As a calibration method, various conventionally known methods can be adopted as exemplified in the following 1) and 2).
1) An image obtained by capturing an object such as a white line drawn on the ground G on which the backhoe 10 is installed or a box measure vertically installed on the ground G with each camera 46 is displayed on the screen. The optical axis of each camera 46 is adjusted so as to match the reference line so that an overhead image can be obtained accurately.
2) Predetermined parameters (to generate an overhead image) used in the overhead image generation process in the overhead image generation means 52 so that an overhead image can be accurately obtained based on the obtained images of the objects captured by the cameras 46. (Parameters related to camera axes necessary for this) are adjusted by software.

It is assumed that the backhoe 10 is traveling on the ground G at the work site.
The tilt angle detection means 54 detects the tilt angle θ at which the backhoe 10 tilts with respect to the ground G based on the images obtained from the cameras 56 (step S10).
The overhead image correction unit 56 determines whether or not the change amount Δθ / Δt of the tilt angle θ per unit time Δt exceeds a predetermined threshold θth (step S12).
When it is determined that the change amount Δθ / Δt has exceeded the threshold value θth, the overhead image correction unit 56 performs the overhead view in a state where the backhoe 10 is in the posture of the ground G on the ground G based on the inclination angle θ. The overhead image generated by the overhead image generation means 52 is corrected so that an image is obtained (step S14).
Then, the bird's-eye view image correction unit 56 determines whether or not the absolute value of the change amount Δθ / Δt per unit time Δt of the inclination angle θ is equal to or less than the threshold value θth (step S16).
When it is determined that the absolute value of the change amount Δθ / Δt is equal to or smaller than the threshold θth, the overhead image correction unit 56 cancels the overhead image correction processing (step S18), and returns to step S10.
On the other hand, when it is determined in step S12 that the absolute value of the change amount Δθ / Δt does not exceed the threshold θth, the overhead image correction process by the overhead image correction unit 56 is not performed, and the process returns to step S10.
If it is determined in step S16 that the absolute value of the change amount Δθ / Δt is not equal to or smaller than the threshold θth, the overhead image correction unit 56 returns to step S14 and maintains the overhead image correction process.

As described above, according to this embodiment, the backhoe 10 on the ground G is inclined according to the inclination of the ground G based on the inclination angle θ at which the backhoe 10 (construction machine) generated during traveling is inclined with respect to the ground G. The overhead image is corrected so that the overhead image in the posture is obtained.
Therefore, even if the backhoe 10 is inclined with respect to the ground G by passing through the local unevenness of the ground G, and the imaging range of each camera 46 is temporarily changed, the inclination angle of the backhoe 10 is This is advantageous in displaying an accurate overhead image without being affected by θ.
In particular, the range of the bird's-eye view image covering the entire circumference of the backhoe 10 is extremely wide, and the influence of the change in the imaging range of each camera 46 on the accuracy of the bird's-eye view image is significant. Since the backhoe 10 is greatly inclined and the amount of change in the imaging range of each camera 46 is large, the influence of the change in the imaging range on the accuracy of the overhead image is suppressed. This is advantageous.

Further, the overhead image correction means 56 corrects the overhead image when the absolute value of the change amount Δθ / Δt per unit time of the tilt angle θ exceeds the threshold θth, and the absolute value of the change amount Δθ / Δt is calculated. The correction of the overhead image is canceled when the threshold θth or less is reached.
Therefore, the overhead image correction process is performed when the degree of unevenness of the ground G on which the backhoe 10 travels is large, and the overhead image correction process is not performed when the degree of unevenness is small. This is advantageous in reducing the burden to be borne.
In addition, after the backhoe 10 overcomes the unevenness of the ground G and assumes the posture of the ground G as inclined, the correction processing of the overhead image is canceled, which is advantageous in improving the accuracy of the overhead image.

In addition, the tilt angle detection unit 54 extracts feature points included in the captured image around the backhoe 10 obtained in time series from at least two cameras 46 out of the plurality of cameras 46, and the features extracted in time series. A feature point movement amount detection unit 54A that detects the movement amount of the three-dimensional position of the feature point based on the correspondence between the points, and a detection angle calculation unit 54B that calculates the inclination angle θ based on the movement amount of the feature point. It was supposed to be.
Therefore, a dedicated sensor for detecting the inclination angle θ is not necessary, and therefore, the number of parts can be reduced, which is advantageous in reducing the cost.

As the inclination angle detecting means 54, the inclination angle θ of the construction machine is obtained by integrating the three-axis acceleration sensor provided on the lower traveling body 12 of the backhoe 10 and the three-axis angular velocity output from the three-axis angular velocity sensor. You may use the integration means to calculate.
In this case, complicated image processing (software processing) is not required as compared with the case where the feature point moving amount detection means 54A and the detection angle calculation means 54B are used as the inclination angle detection means 54, so that the inclination angle θ is detected. Time can be shortened, which is advantageous in reducing the processing time required for the overhead image correction processing.

In the present embodiment, a case has been described in which the control unit 48 of the construction machine side unit 34 realizes the overhead image generation means 52, the inclination angle detection means 54, and the overhead image correction means 56.
However, the captured images from the respective cameras 46 are respectively transmitted to the remote operation unit 32, and the overhead view image generation means 52, the inclination angle detection means 54, and the overhead view image correction means 56 are realized by the control unit 40 of the remote operation unit 32. May be.

  In the present embodiment, the case where four cameras 46 are provided in the construction machine side unit 34 has been described. However, the number of cameras 46 is not limited as long as an overhead image can be generated.

Moreover, although this embodiment demonstrated the case where the backhoe 10 was a thing which can be operated remotely, the backhoe 10 may board | work and operate directly. In this case, the display means may be provided so that an operator operating the backhoe 10 can visually recognize the display means. For example, a display 4814 of the control unit 48 may be used as the display means.
Further, the construction machine is not limited to the backhoe 10, and may be various conventionally known construction machines such as a mobile crane and a bulldozer.

10 Backhoe (construction machine)
46 camera 4602 front camera 4604 rear camera 4606 left camera 4608 right camera 48 control unit 52 overhead image generation means 54 tilt angle detection means 54A feature point movement amount detection means 54B detection angle calculation means 56 overhead image correction means

Claims (2)

A plurality of cameras mounted on construction machines that are operated unattended by remote control ;
An overhead image generation means for generating an overhead image of the entire circumference of the construction machine based on the captured image around the construction machine obtained from the plurality of cameras;
An overhead image display device for construction machinery, comprising: a display means for displaying the overhead image generated by the overhead image generation means;
The plurality of cameras include a front camera that images the front of the construction machine, a rear camera that images the rear of the construction machine, a left camera that images the left side of the construction machine, and a right side of the construction machine. It is configured to obtain an image over the entire circumference including the front, rear, right, and left of the construction machine without a blind spot by including a right camera that captures
The front camera, the rear camera, the left camera, and the right camera are attached to the construction machine so that the optical axes of the cameras can be adjusted around a vertical axis and a horizontal axis,
An inclination angle detecting means for detecting an inclination angle θ at which the construction machine generated during traveling is inclined with respect to the ground;
Overhead image correction means for correcting the overhead image so as to obtain an overhead image in a state in which the construction machine is in a tilted position of the ground on the ground based on the inclination angle θ. ,
The tilt angle detecting means includes
Extracting feature points included in a captured image around the construction machine obtained in time series from at least two of the plurality of cameras, and based on the correspondence relationship of the feature points extracted in time series Feature point movement amount detection means for detecting the movement amount of the three-dimensional position of the feature point;
Detection angle calculation means for calculating the tilt angle θ based on the amount of movement of the feature point,
The angle about the vertical axis and the angle about the horizontal axis of the optical axis of the camera is an image obtained by capturing an image of an object installed with the camera on the basis of the ground on which the construction machine is installed. It has been adjusted to match the reference line displayed above,
The parameters used for the overhead image generation processing in the overhead image generation means are adjusted based on an image obtained by capturing the object with the camera.
An overhead image display apparatus for construction machines, characterized in that The overhead image correction means determines whether or not the absolute value of the amount of change per unit time of the tilt angle θ exceeds a predetermined threshold, and when the absolute value of the amount of change exceeds the threshold Correcting the overhead image and canceling the correction of the overhead image when the absolute value of the amount of change is equal to or less than the threshold;
The overhead image display apparatus for construction machines according to claim 1.

Images