JP2015075426A - Method for calibrating cameras for overhead image display unit mounted on construction machine and overhead image display unit using results thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately calibrate a plurality of cameras.SOLUTION: Ground lines LG that should match an optical axis of each camera 36 in a plane view are plotted for each camera 36 on a flat ground G on which a backhoe 10 is placed and within a range in which each camera 36 can capture an image. An image captured by one camera 36 selected from a plurality of cameras 36 superimposed on a vertical line LV and a horizontal line LH is displayed on a screen 40 of a display 3814. A leveling rod 42 having a prescribed height is vertically placed on the ground line LG on the ground G. An angle about a vertical axis of the optical axis of the selected camera 36 is adjusted so that an image of the leveling rod 42 is overlapped with the vertical line LV, and an angle about a horizontal axis of the optical axis of the selected camera 36 is adjusted so that an image of an indicator 44 which is displayed at the position of the leveling rod 42 having the same height as that of an imaging device of the selected camera 36 is overlapped with the horizontal line LH on the screen 40.

Description

  The present invention relates to a method for calibrating a lens mounted on a construction machine and an overhead image display apparatus using the result.

In an emergency disaster recovery work, when a remotely operated construction machine is operated remotely, the operator needs to operate it from a safe place sufficiently away from the work place.
Although it is possible to operate visually in places where the construction range is narrow, it is usually possible to remotely display an image from a video camera mounted on a construction machine, an image from a moving camera car, an image from a fixed camera, etc. Construction can be done by operating.
However, these cameras are likely to cause blind spots every time the construction machine moves or turns, which is disadvantageous for remote operation.
On the other hand, driving support that provides a driver with a plurality of cameras that capture the periphery of a passenger car, processes the images supplied from each camera, creates an overhead image as seen from above the passenger car An apparatus is provided (see Patent Document 1).
Therefore, it is conceivable to apply such a technique for creating a bird's-eye view image to construction machines.

JP 2006-121587 A

However, if the directions of the cameras (the direction of the optical axis) are not accurately adjusted with respect to a predetermined direction, the joints when processing and joining the images obtained from the cameras are shifted. There is a concern that the overhead image may be inaccurate.
In particular, since construction machines are larger than passenger cars, the range of the bird's-eye view that covers the entire circumference of the construction machine is extremely wide. The impact on the environment is extremely large.
Therefore, there is a problem of how to perform calibration (calibration) for accurately adjusting the orientations of a plurality of cameras.
The present invention has been made in view of such circumstances, and an object thereof is to provide a camera calibration method in an overhead image display apparatus that is advantageous in performing calibration with high accuracy.

  In order to achieve the above-described object, a camera calibration method for a construction machine overhead image display device according to the present invention is based on a captured image around the construction machine obtained from a plurality of cameras mounted on the construction machine. A camera calibration method in an overhead view image display device for generating and displaying an overhead view image of the entire circumference of a construction machine centering on a construction machine, the reference line to which the optical axes of the cameras should coincide in a plan view On the flat ground on which the construction machine is installed and in a range that can be captured by each camera, and a single camera selected from the plurality of cameras A second step of displaying on the image display means the captured image obtained in the above, a vertical line passing through the center on the screen and extending in the vertical direction of the screen on the screen of the image display means, and the screen A third step of displaying a horizontal line passing through the center of the vertical line and perpendicular to the vertical line, a fourth step of installing a rod-shaped body having a predetermined height on the ground at a right angle on the reference line, and the image display means A fifth step of adjusting an angle around the vertical axis of the optical axis of the selected camera so that the image of the rod-shaped body displayed on the screen and the vertical line overlap, and imaging of the selected camera The sixth step of displaying an index at the position of the rod-like body having the same height as the element, and the image of the index displayed on the screen of the image display means and the horizontal line on the screen are selected as described above. A seventh step of adjusting an angle of the optical axis of the camera around a horizontal axis, and the steps from the second step to the seventh step are performed for each of the plurality of cameras.

According to the present invention, a reference line that should match the optical axes of each camera in a plan view is placed on a flat ground where a backhoe is installed and within a range that can be captured by each camera. The first step of drawing was performed, and then the second to seventh steps were performed to adjust the angle around the vertical axis and the horizontal axis of the optical axis of the camera.
For this reason, it is advantageous to perform calibration for accurately adjusting the orientation of each camera accurately, minimizing the deviation of the orientation of each camera, and improving the accuracy of the obtained overhead image. It will be advantageous.

It is a side view of a backhoe. It is a top view of a backhoe. It is a block diagram which shows the structure of the remote control system which carries out unmanned operation of a backhoe by remote control. It is a figure which shows an example of a bird's-eye view image. It is a block diagram which shows the structure of the personal computer for calibration to which the some camera was connected. It is a functional block diagram of a personal computer for calibration to which a plurality of cameras are connected. It is a flowchart which shows the procedure of the calibration of the camera in embodiment. It is explanatory drawing of the reference line drawn on the ground in which the backhoe was installed. It is explanatory drawing of the box measure installed on the ground. It is a figure which shows the display content of the screen of a display before the adjustment around the vertical axis of the optical axis of a camera. It is a figure which shows the display content of the screen of a display after the adjustment around the vertical axis of the optical axis of a camera. It is a figure which shows the display content of the screen of a display before adjustment around the horizontal axis of the optical axis of a camera. It is a figure which shows the display content of the screen of a display after the adjustment around the horizontal axis of the optical axis of a camera.

Hereinafter, the present embodiment will be described with reference to the drawings.
First, a construction machine on which the overhead image display device for construction machine to which the method of the present invention is applied is mounted will be described.
In the present embodiment, a case will be described in which a construction machine is a backhoe that is operated unattended by remote control in a dangerous construction place such as an emergency disaster recovery construction.
In this case, the operator operates the backhoe from a safe place sufficiently away from the construction place.

First, the configuration of the backhoe will be described.
As shown in FIGS. 1 and 2, the backhoe 10 includes a lower traveling body 12, an upper swing body 14, a boom 16, an arm 18, and a bucket 20.

The lower traveling body 12 travels on the ground G by the rotation of the crawler 1202.
The upper turning body 14 is provided on the upper part of the lower traveling body 12 so as to be horizontally turnable around a turning axis.
The upper swing body 14 is provided with an operation chamber 1402, and the operation chamber 1402 travels the lower traveling body 12, swings the upper swing body 14, swings the boom 16, swings the arm 18, swings the bucket 20. A plurality of operation devices such as operation levers and operation pedals (not shown) are provided for operating the devices.

The boom 16 is swingably supported by the upper swing body 14 via a support shaft whose base end extends in the horizontal direction.
The arm 18 is swingably supported at the distal end of the boom 16 via a support shaft whose base end extends in the horizontal direction.
The bucket 20 is swingably supported at the distal end of the arm 18 via a support shaft whose base end extends in the horizontal direction.
A boom cylinder 1602 that swings the boom 16 is provided between the upper swing body 14 and the boom 16.
An arm cylinder 1802 that swings the arm 18 is provided between the boom 16 and the arm 18.
A bucket cylinder 2002 that swings the bucket 20 is provided between the arm 18 and the bucket 20.
These boom cylinder 1602, arm cylinder 1802, and bucket cylinder 2002 are hydraulic cylinders.
Therefore, the boom 16 is swung with respect to the upper swing body 14 by the expansion and contraction of the boom cylinder 1602.
Further, the arm 18 is swung with respect to the boom 16 by the expansion and contraction of the arm cylinder 1802.
Further, the bucket 20 is swung with respect to the arm 18 by the expansion and contraction of the bucket cylinder 2002.

Next, a remote control system for remotely controlling the backhoe 10 will be described.
FIG. 3 is a block diagram showing the configuration of the remote control system.
The remote control system 30 includes a remote operation unit 32 and a construction machine side unit 34.
The remote control unit 32 is provided at a location away from the work site where the backhoe 10 performs work.
The remote operation unit 32 includes an operation unit 3202, a display unit 3204, a control unit 3206, and a communication unit 3208.
The operation unit 3202 is operated by an operator for performing remote control of the construction machine, and includes, for example, a joystick and an operation switch.
The display unit 3204 displays an overhead image as shown in FIG. 4 transmitted from the construction machine side unit 34, and includes a display such as a liquid crystal display device.
The control unit 3206 generates an operation command corresponding to the operation performed on the operation unit 3202, supplies the operation command to the communication unit 3208, and accepts an overhead image transmitted from the construction machine side unit 34 via the communication unit 3208, and displays the display unit 3204. To supply.
The communication unit 3208 communicates an operation command and an overhead image with the communication unit 3406 of the construction machine side unit 34.
These two communication units 3208 and 3406 are connected by a wireless line.
As the wireless line, various conventionally known wireless lines such as SS wireless can be used, and the type, configuration, and system of the wireless line are arbitrary. Further, a plurality of types of wireless lines may be combined, or one or more relay stations may be provided.

The construction machine side unit 34 is provided on the backhoe 10 and includes a drive unit 3402, a plurality of cameras 36, a control unit 3404, and a communication unit 3406.
The drive unit 3402 drives the operation devices of the backhoe 10, and various conventionally known actuators such as an air cylinder and a motor can be used.
The plurality of cameras 36 includes a front camera 3602, a rear camera 3604, a left camera 3606, and a right camera 3608.

As shown in FIGS. 1 and 2, the front camera 3602 is mounted on the front part of the upper swing body 14 and images the front of the backhoe 10.
The rear camera 3604 is mounted on the rear part of the upper swing body 14 and images the rear of the backhoe 10.
The left camera 3606 is mounted on the left side of the upper swing body 14 and images the left side of the backhoe 10.
The right camera 3608 is mounted on the right side of the upper swing body 14 and images the right side of the backhoe 10.
In the present embodiment, the angle of view of each camera 36 is 180 degrees.
Note that the angle of view of each camera 36 may be less than 180 degrees as long as a bird's-eye view image described later can be generated. However, by setting the angle of view of each camera 36 to 180 degrees and installing the cameras 36 on the front, rear, left, and right sides of the upper swing body 14 as described above, images around the entire periphery of the upper swing body 14 (backhoe 10) can be obtained. It can be obtained efficiently and without generating blind spots, which is advantageous in accurately generating a bird's-eye view image as shown in FIG.
Each camera 36 is attached to the upper swing body 14 via a platform (not shown) so that the optical axis of each camera 36 can be adjusted around the vertical axis and the horizontal axis.

The control unit 3404 receives the operation command transmitted from the remote operation unit 32 via the communication unit 3406, and operates the operation devices of the backhoe 10 by controlling the drive unit 3402 based on the operation command. Is.
Further, the control unit 3404 generates an overhead view image (FIG. 4) of the entire circumference of the backhoe 10 around the backhoe 10 based on the captured images around the backhoe 10 supplied from each camera 36, and communicates these captured images. The data is transmitted to the communication unit 3208 of the remote operation unit 32 via the unit 3406.

In addition, the generation of the overhead view image by the control unit 3404 is performed by combining viewpoint conversion images obtained by performing viewpoint conversion processing on captured images obtained from the respective cameras 36. Such viewpoint conversion processing and viewpoint conversion images are performed. As the combination processing, various conventionally known methods can be employed.
Therefore, in the present embodiment, the overhead view image display apparatus includes the display unit 3204, the control unit 3206, the communication unit 3208 of the remote operation unit 32, the communication unit 3406, the control unit 3404, and the cameras 36 of the construction machine side unit 34. Is configured.

In the present embodiment, the case where the overhead view image is generated by the control unit 3404 of the construction machine side unit 34 has been described. However, the captured image from each camera 36 is transmitted to the remote operation unit 32, and the remote operation unit 32. The control unit 3206 may generate a bird's-eye view image based on these captured images.
In the present embodiment, the case where four cameras 36 are provided in the construction machine side unit 34 has been described. However, the number of cameras 36 is not limited as long as an overhead image can be generated. One or a plurality of monitoring cameras are provided, a monitoring image captured by the monitoring camera is transmitted to the remote operation unit 32, and the monitoring image is displayed on the display unit 3204 of the remote operation unit 32. At that time, the direction of the monitoring camera may be fixed in advance, or the direction of the monitoring camera may be changed by remote control.

Next, a camera calibration method for the overhead image display device for construction machine according to the present embodiment will be described.
When calibrating each camera 36, a personal computer for calibration is prepared, and each camera 36 is connected to the personal computer.
As shown in FIG. 5, the personal computer 38 includes a CPU 3802, a ROM 3804, a RAM 3806, a hard disk device 3808, a keyboard 3810, a mouse 3812, a display 3814, and an input / output interface connected via an interface circuit (not shown) and a bus line 3801. 3816 and the like.
A ROM 3804 stores a predetermined control program and the like, and a RAM 3806 provides a working area.
The hard disk device 3808 stores a program for realizing functions necessary for camera calibration.
A keyboard 3810 and a mouse 3812 accept operation inputs from the operator.
The display 3814 displays the captured image supplied from each camera 36 on a screen, and is composed of, for example, a liquid crystal display device. In the present embodiment, the display 38014 constitutes the image display means 38A (FIG. 6).
The input / output interface 3816 exchanges data with peripheral devices. In the present embodiment, the input / output interface 3816 accepts captured images from the cameras 36.

As shown in FIG. 6, the personal computer 38 implements marker generating means 38B and image control means 38C by the CPU 3802 executing the program.
The marker generation unit 38B generates a marker including a vertical line, a horizontal line, and a center circle displayed on the screen of the display 3814 and supplies the marker to the image control unit 38C.
The image control unit 38 </ b> C displays a captured image of one camera 36 selected from captured images supplied from each camera 36 in accordance with an operation of the keyboard 3810 or the mouse 3812 on the display 3814.
In addition, the image control unit 38 displays the marker supplied from the marker generation unit 38B on the display 3814 so as to overlap the captured image supplied from each camera 36.
Further, the image control unit 38C enlarges the display magnification (zoom) of the image display unit 38A in accordance with the operation of the keyboard 3810 or the mouse 3812 and displays the captured image.

As shown in FIG. 10, the screen 40 of the display 3814 has a rectangular shape.
The vertical line LV constituting the marker passes through the center Pc on the screen 40 of the display 3814 and extends in the vertical direction of the screen 40.
The horizontal line LH constituting the marker passes through the center Pc on the screen 40 of the display 3814 and is orthogonal to the vertical line LV.
A center circle C constituting the marker is a perfect circle surrounding a point where the vertical line LV and the horizontal line LH intersect.

Next, a specific procedure for camera calibration of the overhead image display apparatus for construction machines will be described with reference to the flowchart of FIG.
First, the backhoe 10 is installed on the flat ground G, and as shown in FIG. 8, the reference ground LG where the optical axes of the cameras 36 should coincide with each other in a plan view is displayed on the flat ground on which the backhoe 10 is installed. Each camera 36 is drawn on G and in a range that can be imaged by each camera 36 (step S10: first step).
FIG. 8 shows a case where a grid-like reference line LG extending in the front-rear direction and the left-right direction of the backhoe 10 is drawn on the ground G. The reference line LG corresponds to each of the four cameras 36. Just draw each book.

Next, the first operator operates the personal computer 38 to display the captured image obtained by one camera 36 selected from the plurality of cameras 36 on the display 3814 (step S12: second step). ).
Then, the marker generation unit 38B and the image control unit 38C display on the screen 40 of the display 3814 the captured image in which the vertical line LV, the horizontal line LH, and the center circle C are overlapped as shown in FIG. S14: Third step).
In FIGS. 10 and 11, only the reference line LG0 to which the optical axis of the selected camera 36 should coincide and the other reference line LG parallel to the reference line LG0 are shown in order to simplify the drawings. The reference lines LG orthogonal to these reference lines LG0, LG are not shown.

Next, as shown in FIG. 9, the second worker installs a box measure 42 (staff) as a rod-shaped body having a predetermined height on the ground G on the reference line LG at a right angle (step S16: Fourth step). The position where the box measure 42 is installed may be a position where the box measure 42 can be imaged by the selected camera 36.
In addition, since the scale 4202 which shows height (length) is displayed on the surface, the box measure 42 is advantageous when performing the operation | work which displays the parameter | index 44 in step S20 mentioned later easily and reliably.
By installing the box measure 42 at a right angle on the ground G, the screen 40 of the display 3814 has a box measure 42 whose extending direction coincides with the reference line LG0 and a vertical line LV, as shown in FIG. Although displayed, at this stage, the extending direction of the box measure 42 does not match the vertical line LV.

  Next, as shown in FIG. 11, the first worker observes the screen 40 of the display 3814 so that the image of the box measure 42 displayed on the screen 40 of the display 3814 and the vertical line LV overlap each other. Then, the angle around the vertical axis of the optical axis of the selected camera 36 is adjusted (step S18: fifth step).

Next, as shown in FIG. 9, the second worker displays the index 44 at the position of the box measure 42 having the same height as the imaging device of the selected camera 36 (step S20: sixth step). . Then, the first operator operates the personal computer 38 to enlarge and display (zoom) the screen 40 of the display 3814 as shown in FIG. 12 (step S22: sixth step).
As a result, the box measure 42 and the index 44 enlarged and displayed on the screen 40 of the display 3814 are displayed.
At this time, since the center circle C is displayed superimposed on the captured image and the screen 40 is enlarged, the positional relationship between the index 44 and the horizontal line LH is easily seen.
By displaying the indicator 44 on the box measure 42, as shown in FIG. 12, the box measure 42 and the horizontal line LH are displayed on the screen 40 of the display 3814. At this stage, the indicator 44 and the horizontal line LH are displayed. (The position in the vertical direction of the screen 40) does not match.

Next, as shown in FIG. 13, the first worker selects the optical axis of the camera 36 selected so that the image of the index 44 displayed on the screen 40 of the display 3814 and the horizontal line LH on the screen 40 overlap. The angle around the horizontal axis is adjusted (step S24: seventh step).
Thereby, the angle around the vertical axis and the angle around the horizontal axis of the optical axis of the selected camera 36 are adjusted, and the calibration of the selected camera 36 is completed.

Next, it is determined whether or not there is a camera 36 to be calibrated (step S24). If there is a camera 36 to be adjusted, the process returns to step S12 and the same processing (steps S12 to S24: first step). To the seventh step).
When the calibration of all the cameras 36 is completed, the calibration process is terminated.

As described above, according to the present embodiment, the reference line LG to which the optical axes of the cameras 36 should coincide in a plan view is displayed on the flat ground G on which the backhoe 10 is installed, and each camera. An image captured by one camera 36 selected from a plurality of cameras 36 and a vertical line LV and a horizontal line LH are drawn on the screen 40 of the display 3814 in a range that can be captured by the camera 36. A box measure 42 having a predetermined height was placed on the ground G on the reference line LG at a right angle. Then, the angle of the optical axis of the camera 36 selected so that the image of the box measure 42 and the vertical line LV overlap each other is adjusted, and the box measure 42 having the same height as the image sensor of the selected camera 36 is adjusted. The angle around the horizontal axis of the optical axis of the camera 36 selected so that the image of the index 44 displayed at the position of the index 44 overlaps the horizontal line LH on the screen 40 is adjusted.
Therefore, it is advantageous in performing calibration (calibration) for accurately adjusting the orientation of each camera 36 with high accuracy.
Therefore, a construction machine having a size larger than that of a passenger car or the like has a very wide range of a bird's-eye view image covering the entire circumference of the construction machine, and the deviation of the orientation of each camera 36 gives the accuracy of the bird's-eye view image. Although the influence is extremely large, the deviation of the orientation of each camera 36 can be minimized, which is advantageous in improving the accuracy of the obtained overhead view image.

  In the present embodiment, since the backhoe 10 is a construction machine that is operated unattended by remote control, an image from a video camera mounted on the construction machine, an image from a moving camera car, a fixed camera, Compared with remote control based on images, etc., the occurrence of blind spots in captured images caused by the movement and turning of construction machinery can be minimized, which is advantageous in improving workability of remote control. .

In the present embodiment, the case where the construction machine is remotely operable has been described, but the construction machine may be directly operated by an operator. In this case, the image display means 38 </ b> A may be provided so that an operator who operates the construction machine can view the overhead view image.
Further, the construction machine is not limited to the backhoe 10, and may be various conventionally known construction machines such as a mobile crane and a bulldozer.

10 Backhoe (construction machine)
36 Camera 3602 Front Camera 3604 Rear Camera 3606 Left Camera 3608 Right Camera 38A Image Display Means 40 Screen Pc Center LV Vertical Line LH Horizontal Line C Center Circle G Ground LG, LG0 Reference Line 42 Box Measure (Bar-Shaped Body)
44 indicators

Claims (7)

The camera of the overhead image display device that generates and displays an overhead image of the entire circumference of the construction machine based on the captured images around the construction machine obtained from a plurality of cameras mounted on the construction machine. A calibration method comprising:
A reference line that should coincide with the optical axis of each camera in a plan view is drawn for each camera on a flat ground on which the construction machine is installed and within a range that can be captured by each camera. A first step;
A second step of displaying on the image display means a captured image obtained by one camera selected from the plurality of cameras;
Third step of displaying on the screen of the image display means a vertical line passing through the center on the screen and extending in the vertical direction of the screen and a horizontal line passing through the center on the screen and perpendicular to the vertical line. When,
A fourth step of installing a rod-shaped body of a predetermined height on the ground at a right angle on the reference line;
A fifth step of adjusting an angle around the vertical axis of the optical axis of the selected camera so that the image of the rod-shaped body displayed on the screen of the image display means and the vertical line overlap;
A sixth step of displaying an indicator at the location of the rod-like body having the same height as the imaging device of the selected camera;
A seventh step of adjusting an angle around the horizontal axis of the optical axis of the selected camera so that an image of the index displayed on the screen of the image display means and a horizontal line on the screen overlap.
Carrying out the second step to the seventh step for each of the plurality of cameras;
A camera calibration method for an overhead image display apparatus for construction machinery, characterized in that: The plurality of cameras are:
A front camera mounted on the front of the construction machine and imaging the front of the construction machine;
A rear camera mounted on the rear of the construction machine and imaging the rear of the construction machine;
A left camera that is mounted on the left side of the construction machine and images the left side of the construction machine;
A right camera mounted on the right side of the construction machine and imaging the right side of the construction machine;
The angle of view of each camera is 180 degrees.
The camera calibration method of the overhead image display device for construction machines according to claim 1. The third step displays a central circle surrounding a point where the vertical line and the horizontal line intersect on the screen of the display means.
The camera calibration method for the overhead image display device for construction machines according to claim 1 or 2. The sixth step displays an enlarged screen of the display means.
The camera calibration method for a construction machine overhead image display device according to any one of claims 1 to 3. The rod-like body is a box measure.
The camera calibration method for an overhead image display device for a construction machine according to any one of claims 1 to 4. The construction machine is operated unattended by remote control,
The camera calibration method for an overhead image display device for a construction machine according to any one of claims 1 to 5. A plurality of cameras are calibrated by the camera calibration method of the overhead image display device for construction machine according to any one of claims 1 to 6.
A bird's-eye view image display apparatus characterized by

Images