US20130155240A1

US20130155240A1 - Work vehicle periphery monitoring apparatus

Info

Publication number: US20130155240A1
Application number: US13/818,323
Authority: US
Inventors: Shinji Mitsuta; Shigeru Harada; Tomikazu Tanuki; Eishin Masutani; Yukihiro Nakanishi; Takeshi Kurihara; Dai Tsubone; Masaomi Machida
Original assignee: Komatsu Ltd
Current assignee: Komatsu Ltd
Priority date: 2011-06-07
Filing date: 2012-05-24
Publication date: 2013-06-20
Also published as: US20170120823A1; AU2012268486A1; CA2808260A1; CN103079891A; JP5597596B2; JP2012254660A; US9956915B2; CN103079891B; WO2012169362A1; CA2808260C

Abstract

A periphery monitoring apparatus for a work vehicle is adapted to display an image of surroundings of the work vehicle on a display apparatus. The periphery monitoring apparatus includes a camera and a display control section. The camera is disposed below a vehicle body of the work vehicle, the camera being configured to acquire a camera image of the surroundings of the work vehicle. The display control section is configured to display the camera image acquired using the camera and a vehicle body outer edge line on the display apparatus. The vehicle body outer edge line indicates an outer edges of the vehicle body which is vertically projected on a ground surface in the camera image.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No. 2011-127478 filed on Jun. 7, 2011, the disclosure of which is hereby incorporated herein by reference in its entirety.

TECHNICAL FIELD

The present invention relates to a periphery monitoring apparatus, and in particular, to a periphery monitoring apparatus which displays a camera image of the surroundings of a work vehicle on a display apparatus.

BACKGROUND ART

As extra-large work vehicles, dump trucks which are used for carrying crushed rock at a mine or the like are known. In dump trucks of this type, it is difficult for a driver to grasp the circumstances in the surroundings using side mirrors and the like since the vehicle width is remarkably wide in comparison with a typical truck or the like, and in addition, the length from the front to back is long.
Therefore, in Hitachi Construction Machinery Co., Ltd. press release “Joint Development of ‘Overview Monitor System’ with Clarion Co., Ltd.”, (online), Jan. 24, 2011, (retrieved on Jun. 1, 2011), link <URL: http://www.hitachikenki.co.jp/news/press/PR20110121113934973.html>, a periphery monitoring apparatus is proposed where bird's-eye images for the front, left, and right of the work vehicle and a camera image of the rear of the work vehicle are displayed on a display apparatus with an object of a driver simply grasping the circumstances in the periphery. Specifically, the periphery monitoring apparatus of non patent literature 1 is provided with three cameras which video the front, left, and right of the work vehicle and one rear camera which videos the rear of the work vehicle. The rear camera is disposed below the vehicle body which has a vessel and the ground surface at the rear of the work vehicle appears in the camera image.

SUMMARY

However, in the camera image of non patent literature 1, grasping of the positional relationship of the vehicle body and objects in the periphery of the vehicle body (for example, a person, a vehicle, a building, or the like) is not easy since the vehicle body exists upward of the rear camera. That is, it is difficult to determine using the camera image if a person has entered the space below the vehicle body, to what extent that the vehicle body has approached a building, or the like.
The present invention has been carried out in consideration of the circumstances described above and has an object of providing a periphery monitoring apparatus where it is possible to easily grasp the positional relationship of the vehicle body and objects in the periphery of the vehicle body using a camera image.
A periphery monitoring apparatus for a work vehicle of a first aspect displays an image of the surroundings of the work vehicle on a display apparatus, the periphery monitoring apparatus includes a camera being disposed below a vehicle body of the work vehicle, the camera being configured to acquire a camera image of the surroundings of the work vehicle; and a display control section being configured to display the camera image acquired using the camera and a vehicle body outer edge line on the display apparatus, the vehicle body outer edge line indicating an outer edges of the vehicle body which is vertically projected on a ground surface in the camera image.
According to the periphery monitoring apparatus for a work vehicle of the first aspect, it is possible for the driver to easily grasp the positional relationship of the vehicle body which is positioned above the camera and objects in the periphery of the vehicle body (for example, a person, a vehicle, a building, or the like) with the vehicle body outer edge line as a reference. As a result, it is possible to easily determine using the camera image, for example, if a person has entered the space below the vehicle body, to what extent that the vehicle body has approached a building, or the like. As a result, it is possible for the driver to simply grasp the circumstances in the periphery of the work vehicle.
A periphery monitoring apparatus for a work vehicle of a second aspect is related to the first aspect, the display control section is configured to display a warning line on an outside of the vehicle body outer edge line, the warning line indicating a position with a predetermined distance from the vehicle body outer edge line on the ground surface in the camera image.
According to the periphery monitoring apparatus for a work vehicle of the second aspect, it is possible for the driver to more easily grasp the positional relationship of the vehicle body and objects in the periphery of the vehicle body since to what extent objects in the periphery of the vehicle body are separated from the vehicle body is shown along with the vehicle body outer edge line in a stepped manner.
A periphery monitoring apparatus for a work vehicle of a third aspect is related to the first or the second aspect, the vehicle body has a frame and a vessel disposed on the frame, a rear edge portion of the vessel is included in an imaging range of the camera, and the vehicle body outer edge line shows the rear edge portion of the vessel which is vertically projected on the ground surface in the camera image.
According to the periphery monitoring apparatus for a work vehicle of the third aspect, it is possible for the driver to more clearly grasp the positional relationship of the vessel and objects in the periphery of the vessel since it is possible for the rear edge portion of the vessel to be directly confirmed in the camera image.
A periphery monitoring apparatus for a work vehicle of a fourth aspect displays an image of the surroundings of the work vehicle on a display apparatus and includes a plurality of cameras being mounted on a vehicle body of the work vehicle, each of the plurality of cameras being configured to acquire a camera image of the surroundings of the work vehicle; and a display control section being configured to lines up a bird's-eye image of the surroundings of the work vehicle and a specific camera image out of a plurality of camera images acquired using the plurality of cameras on the camera image, the bird's-eye image being acquired based on the plurality of images, the display control section being configured to display a vehicle body outer edge line in the specific camera image, the vehicle body outer edge line indicating an outer edges of the vehicle body which is vertically projected on a ground surface in the specific camera image.
According to the present invention, it is possible to provide a periphery monitoring apparatus where it is possible to easily determine the positional relationship of the vehicle body and objects in the periphery of the vehicle body using a camera image.

BRIEF DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

FIG. 1 is a perspective diagram illustrating an overall configuration of a dump truck.

FIG. 2 is a block diagram illustrating a configuration of a periphery monitoring apparatus.

FIG. 3 is a planar diagram of a dump truck which illustrates mounting positions of six cameras and two-dimensional videoing ranges.

FIG. 4 is a perspective diagram of a dump truck which illustrates three-dimensional imaging ranges using six cameras.

FIG. 5 is a perspective diagram illustrating an attachment state of six cameras.

FIG. 6 is a schematic diagram illustrating a configuration of a monitor.

FIG. 7 is a schematic diagram illustrating a configuration of a monitor.

DESCRIPTION OF EMBODIMENTS

Below, an embodiment of the present invention will be described by referencing the diagrams. Here, in the description below, “front, back, left, and right” are terms with an operator who is sitting in a driver's seat as a reference, and “vehicle width direction” has the same meaning as “left and right direction”.

Overall Configuration of Dump Truck

FIG. 1 is a perspective diagram illustrating an overall configuration of a dump truck 1 which is provided with a periphery monitoring apparatus according to an embodiment of the present invention. The dump truck 1 is a self-propulsion type of extra-large work vehicle which is used in mining operations and the like.
The dump truck 1 is mainly provided with a vehicle body frame 2, a cab 3, a vessel 4, two front wheels 5, two rear wheels 6, and a base 7. In addition, the dump truck 1 is provided with a periphery monitoring apparatus 10 which displays an image of the surroundings of the vehicle on a monitor. The configuration of the periphery monitoring apparatus 10 will be described later.
The vehicle body frame 2 supports a diesel engine, a driving mechanism such as a transmission unit, other auxiliary units, and the like which are not shown. In addition, the front wheels 5 (only the right front wheel is shown in the diagram in FIG. 1) are supported in the left and right of the front portion of the vehicle body frame 2, and the rear wheels 6 (only the right rear wheel is shown in the diagram in FIG. 1) are supported in the left and right of the rear portion of the vehicle body frame 2. The vehicle body frame 2 has a lower deck 2A at a side which is close to the ground surface and has an upper deck 2B above the lower deck 2A. Two movable ladders 2C are provided between the lower deck 2A and the ground surface and a sloping ladder 2D is provided between the lower deck 2A and the upper deck 2B. Hand rails with a palisade shape are fixed in the left and right of the front portion, the side portion, and a portion of the back portion on the upper deck 2B.
The cab 3 is disposed to be lateralized towards the left side from the center in the vehicle width direction above the upper deck 2B. A driver's seat, a shift lever, a controller, a monitor, a handle, an acceleration pedal, a brake pedal, and the like are provided in the cab 3. As will be described later, the controller, the monitor, and the shift level configure a portion of the periphery monitoring apparatus 10.
The vessel 4 is a container for hauling heavy materials such as crushed rocks. A bottom portion at the rear side of the vessel 4 is connected to a rear edge portion of the vehicle body frame 2 so as to be able to rotate via a rotation pin (which is not shown). Due to this, it is possible for the vessel 4 to take an erect posture, where loading materials are discharged by the front portion being rotated upward using an actuator such as a hydraulic cylinder which is not shown, and a hauling posture where the front portion is position at an upper portion of the cab as shown in FIG. 1.
The base 7 is disposed above the upper deck 2B. The base 7 is a power supply apparatus for positioning a pantograph which receives electrical power from a bird's-eye wire such as a trolley.
Here, the vehicle body frame 2 and the vessel 4 configure the “vehicle body” of the dump truck 1 in the embodiment.
Configuration of Periphery Monitoring Apparatus
FIG. 2 is a block diagram illustrating a configuration of the periphery monitoring apparatus 10 which is provided in the dump truck 1.
The periphery monitoring apparatus 10 has six cameras 11 to 16, six frame memories 11A to 16A, an operator switch 20, a monitor 30, and a display control section 40.
Cameras
FIG. 3 is a planar diagram of the dump truck 1 which illustrates mounting positions of the six cameras 11 to 16 and two-dimensional videoing ranges. FIG. 4 is a perspective diagram of the dump truck 1 which illustrates three-dimensional imaging ranges using the six cameras 11 to 16. FIG. 5 is a perspective diagram illustrating an attachment state of the six cameras 11 to 16. The six cameras 11 to 16 are mounted on the outside of the dump truck 1 and acquire images of the surroundings of the dump truck 1.
A first camera 11 is disposed at an upper edge portion of the sloping ladder 2D and is fixed toward the front of the vehicle. A first videoing range 11R of the first camera 11 is the front of the vehicle.
A second camera 12 is disposed at a right edge portion on a front side surface of the upper deck 2B and is fixed toward the front right diagonal of the vehicle. A second videoing range 12R of the second camera 12 is the front right diagonal of the vehicle.
A third camera 13 is disposed at a position which is left and right symmetrical with the second camera 12, that is, a left edge portion on a front side surface of the upper 2B and is fixed toward the front left diagonal of the vehicle. A third videoing range 13R of the third camera 13 is the front left diagonal.
A fourth camera 14 is disposed at a front edge portion on a right side surface of the upper deck 2B and is fixed toward the rear right diagonal of the vehicle. A fourth videoing range 14R of the fourth camera 14 is the rear right diagonal.
A fifth camera 15 is disposed at a position which is left and right symmetrical with the fourth camera 14, that is, a front edge portion on a left side of the upper deck 2B and is fixed toward the rear left diagonal of the vehicle. A fifth videoing range 15R of the fifth camera 15 is the rear left diagonal.
A sixth camera 16 is disposed to be beneath the vessel 4 as shown in FIG. 5 and is fixed toward the rear of the vehicle. Specifically, the sixth camera 16 is fixed by a bracket 9 a which is attached to a cross bar 9 above a rear axel 8. The sixth camera 16 is fixed diagonally downward toward the rear and a sixth imaging range 16R of the sixth camera 16 is the rear of the vehicle below the vessel 4. Here, in the embodiment, the vehicle body itself (the rear edge portion of the vessel 4) is included in the sixth imaging range 16R of the sixth camera 16 (refer to FIG. 6) which is different to the imaging range images of the other cameras.
Images of substantially the entire surroundings of the dump truck 1 are acquired using the six cameras 11 to 16 above. Each of the six cameras 11 to 16 respectively transmit camera image data, which indicates the camera image which is videoed by the apparatus itself, to the frame memories 11A to 16A.
Frame Memories
The six frame memories 11A to 16A are connected to the six cameras 11 to 16. The six frame memories 11A to 16A temporarily store the camera image data which is received from the six cameras 11 to 16. The six frame memories 11A to 16A transmit the camera image data which has been stored to the display control section 40.
Operator Switch
The operator switch 20 is an operating member for specifying the camera image, which is to be displayed on the monitor 30, out of the camera images from each of the six cameras 11 to 16. It is possible for a desired camera image to the displayed on the monitor 30 by the driver operating the operator switch 20. The operator switch 20 notifies the display control section 40 whether any of the camera images has been selected.
Monitor
FIG. 6 and FIG. 7 are schematic diagrams illustrating a configuration of the monitor 30.
The monitor 30 is disposed in front of the driver's seat in the cab 3. The monitor 30 has a display region 30 a which displays the camera image as shown in FIG. 6 and FIG. 7. Here, FIG. 6 shows an appearance where a camera image 16G of the sixth camera 16 and a bird's-eye image F are displayed in the display region 30 a. FIG. 7 shows an appearance where a camera image 15G of the fifth camera 15 and the bird's-eye image F are displayed in the display region 30 a.
Here, a plurality of camera images may be displayed in the display region 30 a or only the bird's-eye image F may be displayed in the display region 30 a. In this manner, the display of the monitor 30 may be automatically switched according to the movement state of the dump truck 1. For example, it is possible for the camera image of the first camera and the bird's-eye image F to be displayed during forward progression and when stationary and only the camera image 16G of the sixth camera 16 to be displayed during backward progression. Furthermore, the camera image 16G and the bird's-eye image F may be displayed during backward progression and the camera image 16G and another camera image may be displayed during backward progression.
Display Control Section
The display control section 40 selects camera image data which corresponds to the camera image which has been selected based on the notification from the operator switch 20. In addition, the display control section 40 generates bird's-eye image data where the plurality of images are videoed from a predetermined projected plane due to conversion of the coordinates of the camera image data of the six cameras 11 to 16 and which shows the bird's-eye image F which encompasses the entire surroundings of the dump truck 1. The display control section 40 generates two-screen data which corresponds to a two-screen image where the camera image which has been selected and the bird's-eye image F are lined up due to the combination of the camera image data and the bird's-eye image data.
In addition, the display control section 40 generates vehicle body outer edge line data which corresponds to a vehicle body outer edge line BL showing the outer edges of the vehicle body (including the vehicle body frame 2 and the vessel 4) which is vertically projected with regard to the ground surface in the camera image which has been selected. The display control section 40 generates the vehicle body outer edge line data according to distortions of the camera image of a target camera in accordance with a sequence which will be described next.
First, the display control section 40 defines three vectors Ra, Rb, and Rc which indicate the orientation of the target cameras based on the following formulae (1) to (3).
$\begin{matrix} Formulae (1) to (3) \\ Ra = (\begin{matrix} 1 & 0 & 0 \\ 0 & ca & sa \\ 0 & - sa & ca \end{matrix}) & (1) \\ Rb = (\begin{matrix} cb & 0 & - sb \\ 0 & 1 & 0 \\ sb & 0 & cb \end{matrix}) & (2) \\ Rc = (\begin{matrix} cc & sc & 0 \\ - sc & cc & 0 \\ 0 & 0 & 1 \end{matrix}) & (3) \end{matrix}$
Here,

- ca=cos(roll), sa=sin(roll)
- cb=cos(pitch), sb=sin(pitch)
- cc=cos(yaw), sc=sin(yaw)

where

- roll is the deflection angle of the target camera around an axis which is parallel with the front and back direction,
- pitch is the deflection angle of the target camera around an axis which is parallel with the vehicle width direction,
- yaw is the deflection angle of the target camera around an axis which is parallel with the vertical direction, and
- pitch, roll, and yaw are stored in a memory which is built into the display control section 40.

Next, the display control section 40 converts the three-dimensional coordinates (Xr, Yr, Zr) at one point on the outer edge of the vehicle body, which is vertically projected with regard to the ground surface, in a physical coordinates system to three-dimensional coordinates (x, y, z) in a camera coordinates system where the orientation and the position of the target camera is a reference as an Euler angle in the vertical direction—front and back direction—vehicle width direction based on the following formula (4).
$\begin{matrix} Formula (4) \\ (\begin{matrix} x \\ y \\ z \end{matrix}) = RaRbRc ((\begin{matrix} Xr \\ Yr \\ Zr \end{matrix}) - (\begin{matrix} tx \\ ty \\ tz \end{matrix})) & (4) \end{matrix}$
Here, (tx, ty, tz) are coordinates in the physical coordinates system of the target camera.
Next, the display control section 40 converts the three-dimensional coordinates (x, y, z) in the camera coordinates system to two-dimensional coordinates (u, v) in the display region 30 a of the monitor 30 based on the following formulae (5) to (11).
Formulae (5) to (11)
x2=x/z (5)
y2=y/z (6)
r ² =x2² +y2² (7)
x3=x2(1+k1*r ² +k2*r ⁴ +k3*r ⁶)+2*p1*x2*y2+p2(r ²+2*x2²) (8)
y3=y2(1+k1*r ² +k2*r ⁴ +k3*r ⁶)+p1(r ²+2*y2²) +2*p2*x2*y2 (9)
u=fx*x3+cx (10)
v=fy*y3+cy (11)
Here, k1, k2, and k3 are radial direction strain coefficients,
P1 and p2 are shear direction strain coefficients,
fx and fy are camera focal distances, and
cx and cy are camera coordinate systems with the focal direction of the camera as the Z axis, the imaging plane direction as the X axis and the Y axis.
In this manner, the display control section 40 acquires the two-dimensional coordinates (u, v) at one point on the outer edge of the vehicle, which is vertically projected with regard to the ground surface, in the display region 30 a. The display control section 40 generates the vehicle body outer edge line data which corresponds to the vehicle body outer edge line BL with a desired form (a dashed line, a chain line, or the like) by acquiring the two-dimensional coordinates in the same manner with regard to a plurality of points on the outer edges of the vehicle body.
In addition, the display control section 40 generates warning line data which corresponds the a first to a third warning lines L1 to L3 which indicate positions with predetermined distances on the outside from the vehicle body outer edge line BL using the formulae (1) to (11) described above in the same manner. In the embodiment, the first warning line L1 is a line which is 3 m to the outside of the vehicle body outer edge line BL, the second warning line L2 is a line which is 5 m to the outside of the vehicle body outer edge line BL, and the third warning line L3 is a line which is 7 m to the outside of the vehicle body outer edge line BL.
The display control section 40 generates display image data by overlapping the two-screen data which has been generated, the vehicle body outer edge line data which has been generated, and the warning line data which has been generated. The display control section 40 transmits the display image data which has been generated to the monitor 30. Due to this, the vehicle body outer edge line BL, which indicates the outer edges of the vehicle body which is vertically projected with regard to the ground surface, and the first to the third warning lines L1 to L3, which indicates the lines which are 3 m, 5 m, and 7 m more to the outside of the vehicle body than the vehicle body outer edge line BL, are displayed along with the camera images 15G and 16G on the monitor 30 as shown in FIG. 6 and FIG. 7.
Actions and Effects
(1) The periphery monitoring apparatus 10 is provided with the display control section 40 which displays the camera image and the vehicle body outer edge line BL, which indicates the outer edges of the vehicle body (including the vessel 4) which is vertically projected with regard to the ground surface in the camera image, on the monitor 30.
Accordingly, it is possible for the driver to easily grasp the positional relationship of the vehicle body which is positioned above the camera and an object (for example, a person, a vehicle, a building, or the like) in the periphery of the vehicle body with the vehicle body outer edge line as a reference. As a result, it is possible to easily determine using the camera image, for example, if a person has entered the space below the vehicle body, to what extent that the vehicle body has approached a building, or the like. As a result, it is possible for the driver to simply grasp the circumstances in the periphery of the dump truck 1.
(2) The display control section 40 displays the first to the third warning lines L1 to L3, which indicate positions with predetermined distances from the vehicle body outer edge line BL, on the monitor 30.
Accordingly, it is possible for the driver to more easily grasp the positional relationship of the vehicle body and objects in the periphery of the vehicle body since to what extent objects in the periphery of the vehicle body are separated from the vehicle body is shown in a stepped manner with the vehicle body outer edge line BL as a reference.
(3) The ground surface is included in the lower part of the screen in the imaging range 16R of the sixth camera 16 as shown in FIG. 6, and the lower surface side of the rear edge portion of the vessel 4 which configures a portion of the vehicle body is included in an upper part of the screen which opposes the ground surface.
Accordingly, it is possible for the driver to more clearly grasp the positional relationship of the vessel and objects in the periphery of the vessel since it is possible for the rear edge portion of the vessel 4 to be directly confirmed in the camera image 16G.

Other Embodiments

The present invention is not limited to the embodiment as described above and various changes and modifications which do not depart from the scope of the invention are possible.
(A) In the embodiment described above, the display control section 40 displays the first to the third warning lines L1 to L3 on the monitor 30 but the present invention is not limited to this. The display control section 40 may display any of the first to the third warning lines L1 to L3, and in addition, may display four or more warning lines.
(B) In the embodiment described above, the display control section 40 shows the vehicle body outer edge line BL and the first to the third warning lines L1 to L3 with dashed lines but the present invention is not limited to this. The display control section 40 may show the vehicle body outer edge line BL and the first to the third warning lines L1 to L3 with a solid line, a chain line, or the like. In addition, the display control section 40 may show each of the vehicle body outer edge line BL and the first to the third warning lines L1 to L3 with different colors.
(C) In the embodiment described above, the case is described where the vehicle body outer edge line BL is displayed along with the camera image 16G of the sixth camera 16 or the camera image 15G of the fifth camera 15 using FIG. 6 and FIG. 7 but the present invention is not limited to this. The vehicle body outer edge line BL may be displayed along with the camera images of the first to the fourth cameras in a case where the driver specifies any of the first to the fourth cameras 11 to 14.
(D) In the embodiment described above, the dump truck 1 is provided with the base 7 but need not be provided with the base 7.
(E) In the embodiment described above, the dump truck 1 is provided with six cameras 11 to 16 but the present invention is not limited to this. The dump truck 1 may be provided with one to five cameras or may be provided with seven or more cameras.

Claims

1. A periphery monitoring apparatus for a work vehicle adapted to display an image of surroundings of the work vehicle on a display apparatus, the periphery monitoring apparatus comprising:

a camera disposed below a vehicle body of the work vehicle, the camera being configured to acquire a camera image of the surroundings of the work vehicle; and

a display control section configured to display the camera image acquired using the camera and a vehicle body outer edge line on the display apparatus, the vehicle body outer edge line indicating an outer edges of the vehicle body which is vertically projected on a ground surface in the camera image.

2. The periphery monitoring apparatus for a work vehicle according to claim 1, wherein the display control section is configured to display a warning line on an outside of the vehicle body outer edge line, the warning line indicating a position at a predetermined distance from the vehicle body outer edge line on the ground surface in the camera image.

3. The periphery monitoring apparatus for a work vehicle according to claim 1, wherein

the vehicle body has a frame and a vessel disposed on the frame,

a rear edge portion of the vessel is included in an imaging range of the camera; and

the vehicle body outer edge line shows the rear edge portion of the vessel which is vertically projected on the ground surface in the camera image.

4. A periphery monitoring apparatus for a work vehicle adapted to display an image of surroundings of the work vehicle on a display apparatus, the periphery monitoring apparatus comprising:

a plurality of cameras mounted on a vehicle body of the work vehicle, each of the cameras being configured to acquire a camera image of the surroundings of the work vehicle; and

a display control section configured to line up a bird's-eye image of the surroundings of the work vehicle and one of the camera images acquired using the cameras on the display apparatus, the bird's-eye image being acquired based on the camera images, the display control section being further configured to display a vehicle body outer edge line in the one of the camera images, the vehicle body outer edge line indicating an outer edges of the vehicle body which is vertically projected on a ground surface in the one of the camera images.