JP2022072598A - Work vehicle display system and work vehicle display method - Google Patents

Abstract

To provide a work vehicle display system and a work vehicle display method capable of performing accurate work and reducing operator fatigue.SOLUTION: A second complementary image pickup device IS2 takes an image below an image pickup range of a first complementary image pickup device IS2. On a head-mounted display 200, a controller 50 displays a display image in which the blind spot is complemented based on at least one of the complementary image taken by the first complementary image pickup device IS1 and the second complementary image taken by the second complementary image pickup device IS2.SELECTED DRAWING: Figure 7

Description

The present disclosure relates to a work vehicle display system and a work vehicle display method.

In a work vehicle, for example, Japanese Patent Application Laid-Open No. 2008-143701 (Patent Document 1) discloses a technique for improving the front view by using a head-mounted display. In Patent Document 1, a partial image obtained by cutting out a range corresponding to a shield such as a mast for raising and lowering luggage of a forklift is superimposed on an external landscape and displayed on a head-mounted display. This allows the operator to see the image behind the shield at the same time.

Japanese Unexamined Patent Publication No. 2008-143701

However, in a work vehicle such as a hydraulic excavator, work tools such as a bucket move up and down. Therefore, when moving from the normal excavation work to the deep excavation work, the floor surface of the driver's cab may become an obstacle and the excavation surface or the work point (for example, the bucket cutting edge) may not be visible. In such a case, the operator either works without seeing the excavated surface or the like, or leans forward and looks into the excavated surface or the like. If you work without seeing the excavated surface, you will not be able to perform accurate work. In addition, when the operator leans forward and works, the operation becomes unstable and the continuous work makes it easy to get tired.

An object of the present disclosure is to provide a work vehicle display system and a work vehicle display method capable of performing accurate work and reducing operator fatigue.

The display system of the work vehicle of the present disclosure includes a display device, a first complementary image pickup device, a second complementary image pickup device, and a controller. The second complementary imaging device captures images below the imaging range of the first complementary imaging device. The controller displays on the display device a display image in which the blind spot is complemented based on at least one of the supplementary image taken by the first complementary imager and the second complementary image taken by the second complementary imager.

The other work vehicle display system of the present disclosure includes a work machine, a driver's cab, a display device, a complementary image pickup device, and a controller. The driver's cab has a first front pillar and a second front pillar located closer to the working machine than the first front pillar. The display device is attached to the operator of the work vehicle. The controller displays a display image complemented by the blind spot by the second front pillar on the display device based on the complementary image taken by the complementary image pickup device.

Yet another work vehicle display system of the present disclosure includes a driver's cab, a display device, a complementary imaging device, and a controller. The cab has a floor. The display device is attached to the operator of the work vehicle. The controller displays a display image complementing the blind spot by the floor of the driver's cab on the display device based on the complementary image captured by the complementary imaging device.

The display method of the work vehicle of the present disclosure is a display method of a work vehicle including a display device, a first complementary image pickup device, and a second complementary image pickup device. The second complementary imaging device captures images below the imaging range of the first complementary imaging device. The work vehicle display method of the present disclosure has the following steps.

The first complementary image taken by the first complementary imager is acquired. The second complementary image taken by the second complementary imager is acquired. A display image in which the blind spot is complemented based on at least one of the first complementary image and the second complementary image is displayed on the display device.

According to the present disclosure, it is possible to realize a work vehicle display system and a work vehicle display method capable of performing accurate work and reducing operator fatigue.

It is a perspective view which shows schematic structure of the work vehicle in one Embodiment of this disclosure. It is a perspective view which shows the mounting state of the 1st complementary image pickup apparatus. It is a perspective view which shows the attachment state of the 2nd complementary image pickup apparatus. It is a figure which shows the structure of the display system of the work vehicle shown in FIG. It is a figure for demonstrating the normal work and the deep digging work in the work vehicle shown in FIG. It is a perspective view which shows the structure of the head-mounted display shown in FIG. It is a figure which shows an example of the functional block of the display system shown in FIG. It is a flow figure which shows an example of the display method of the work vehicle in one Embodiment of this disclosure. It is a perspective view which shows the state that a work vehicle is doing a deep digging work. FIG. 9 is a diagram showing a state in which a main image captured by the main image pickup device and a second complementary image pickup image of the second complementary image pickup device are superimposed during the work shown in FIG. 9. It is a side view which shows a state that a work vehicle is excavating a slope. It is a figure which shows the state which superimposes the main image pickup image of a main image pickup apparatus, and the 1st complement image pickup image of a 1st complementary image pickup apparatus at the time of work shown in FIG. It is a side view which shows how the work vehicle loads a load on a dump truck. It is a figure which shows the state which superimposes the main image pickup image of a main image pickup apparatus, and the first complement image pickup image of a 1st complementary image pickup apparatus at the time of work shown in FIG. It is a figure which shows another example of the functional block of the display system shown in FIG. It is a flow diagram which shows the other example of the display method of the work vehicle in one Embodiment of this disclosure. It is a figure which shows the mode that the 1st complementary image pickup image of a 1st complementary image pickup apparatus and the 2nd complementary image pickup image of a 2nd complementary image pickup apparatus are integrated, and one integrated image is generated. It is a front view which shows the state of a deep digging work. It is a side view which shows the state of a deep digging work.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.
In the specification and drawings, the same components or corresponding components are designated by the same reference numerals and duplicate description is not repeated. Further, in the drawings, the configuration may be omitted or simplified for convenience of explanation. Further, at least a part of the embodiment and each modification may be arbitrarily combined with each other.

The present disclosure is applicable to work vehicles such as wheel loaders, bulldozers, motor graders, and forklifts, in addition to hydraulic excavators. In the following description, "top", "bottom", "front", "rear", "left", and "right" are based on the operator seated in the driver's seat 4S in the driver's cab 4 shown in FIG. It is the direction that was done.

<Structure of work vehicle>
First, the configuration of the work vehicle of the present embodiment will be described with reference to FIGS. 1 to 3.

FIG. 1 is a perspective view schematically showing a configuration of a work vehicle according to an embodiment of the present disclosure. FIG. 2 is a perspective view showing an attached state of the first complementary imaging device. FIG. 3 is a perspective view showing an attached state of the second complementary imaging device.

As shown in FIG. 1, the hydraulic excavator 100 has a main body 1 and a hydraulically operated working machine 2. The main body 1 has a swivel body 3 and a traveling body 5. The traveling body 5 has a pair of tracks 5Cr and a traveling motor 5M. The hydraulic excavator 100 can travel by rotating the track 5Cr. The traveling motor 5M is provided as a drive source for the traveling body 5. The traveling motor 5M is a hydraulic motor operated by hydraulic pressure. The traveling body 5 may have wheels (tires).

The swivel body 3 is arranged on the traveling body 5 and is supported by the traveling body 5. The swivel body 3 can swivel with respect to the traveling body 5 about the swivel shaft RX. The swivel body 3 has a driver's cab 4 (cab). The operator (occupant) of the hydraulic excavator 100 gets on the driver's cab 4 and operates the hydraulic excavator 100. A driver's seat 4S on which the operator sits is provided in the driver's cab 4. The operator can operate the hydraulic excavator 100 in the driver's cab 4. The operator can operate the working machine 2 in the driver's cab 4, can operate the turning body 3 with respect to the traveling body 5, and can operate the hydraulic excavator 100 by the traveling body 5.

The swivel body 3 has an engine cover 9 and a counterweight provided at the rear of the swivel body 3. The engine cover 9 covers the engine room. An engine unit (engine, exhaust treatment structure, etc.) is arranged in the engine room.

The working machine 2 is supported by the swivel body 3. The working machine 2 has a boom 6, an arm 7, and a bucket 8. The working machine 2 further includes a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12.

The boom 6 is rotatably connected to the main body 1 (traveling body 5 and swivel body 3). Specifically, the base end portion of the boom 6 is rotatably connected to the swivel body 3 with the boom foot pin 13 as a fulcrum.

The arm 7 is rotatably connected to the boom 6. Specifically, the base end portion of the arm 7 is rotatably connected to the tip end portion of the boom 6 with the boom top pin 14 as a fulcrum. The bucket 8 is rotatably connected to the arm 7. Specifically, the base end portion of the bucket 8 is rotatably connected to the tip end portion of the arm 7 with the arm top pin 15 as a fulcrum. The bucket 8 has a plurality of cutting edges 8a.

One end of the boom cylinder 10 is connected to the swivel body 3, and the other end is connected to the boom 6. The boom 6 can be driven with respect to the main body 1 by the boom cylinder 10. By this drive, the boom 6 can rotate in the vertical direction with respect to the swivel body 3 with the boom foot pin 13 as a fulcrum.

One end of the arm cylinder 11 is connected to the boom 6 and the other end is connected to the arm 7. The arm 7 can be driven with respect to the boom 6 by the arm cylinder 11. By this drive, the arm 7 can rotate in the vertical direction or the front-back direction with respect to the boom 6 with the boom top pin 14 as a fulcrum.

One end of the bucket cylinder 12 is connected to the arm 7 and the other end is connected to the bucket link. The bucket 8 can be driven with respect to the arm 7 by the bucket cylinder 12. By this drive, the bucket 8 can rotate in the vertical direction with respect to the arm 7 with the arm top pin 15 as a fulcrum.

The driver's cab 4 mainly has a floor portion FR, a plurality of pillars, a front upper beam BT, and a roof portion RF. The plurality of pillars have, for example, a first front pillar FPL, a second front pillar FPR, a pair of center pillars, and a pair of rear pillars. Each of the plurality of pillars extends upward from the floor FR. The front upper beam BT connects the upper end of the first front pillar FPL and the upper end of the second front pillar FPR. The roof RF is located at the top of a plurality of pillars.

The driver's seat 4S is arranged in the space surrounded by the floor FR, a plurality of pillars, and the roof RF. The driver's seat 4S is located above the floor portion FR and below the roof portion RF. Each of the first front pillar FPL and the second front pillar FPR is arranged in front of the driver's seat 4S. The second front pillar FPR is located closer to the working machine 2 than the first front pillar FPL.

A first complementary image pickup apparatus IS1 and a second complementary imaging apparatus IS2 are attached to the driver's cab 4. Each of the first complementary image pickup apparatus IS1 and the second complementary imaging apparatus IS2 is attached to the second front pillar FPR.

Each of the first complementary image pickup apparatus IS1 and the second complementary imaging apparatus IS2 is arranged outside the cab 4, and images the outside of the cab 4. Each of the first complementary image pickup apparatus IS1 and the second complementary imaging apparatus IS2 is arranged in front of the driver's cab 4 and can take an image in front of the driver's cab 4.

The second complementary image pickup apparatus IS2 is attached below the first complementary imaging apparatus IS1. The second complementary imaging device IS2 images an imaging range below the imaging range of the first complementary imaging device IS1.

As shown in FIG. 2, the first complementary image pickup apparatus IS1 is attached to the driver's cab 4 via the bracket BR1 and the holder CH1. The bracket BR1 is outside the driver's cab 4, and is attached to the upper right corner of the driver's cab 4 with screws or the like. The holder CH1 is attached to the front surface of the bracket BR1 with screws or the like. The first complementary image pickup apparatus IS1 is attached to the holder CH1 with screws or the like. The first complementary image pickup apparatus IS1 is located above the height position of the upper end of the second front pillar FPR.

As shown in FIG. 3, the second complementary image pickup apparatus IS2 is attached to the driver's cab 4 via the bracket BR2 and the holder CH2. The bracket BR2 is outside the cab 4, and is attached to the lower right side of the front surface of the cab 4 with screws or the like. The bracket BR2 has a front extending portion EX1, a downward extending portion EX2, and a mounting portion AP.

The forward extending portion EX1 extends in the front-rear direction in front of the driver's cab 4. The downward extending portion EX2 extends downward from the front end of the anterior extending portion EX1. The mounting portion AP is mounted at the lower end of the downward extending portion EX2.

The holder CH2 is attached to the ends of the attachment portion AP in the left-right direction on the side closer to the working machine by screws or the like. The second complementary image pickup device IS2 is attached to the holder CH2 with screws or the like. The second complementary image pickup apparatus IS2 is located below the height position of the lower end of the second front pillar FPR.

<Display system configuration>
Next, the configuration of the display system of the present embodiment will be described with reference to FIGS. 4 and 5.

FIG. 4 is a diagram showing a configuration of a display system for the work vehicle shown in FIG. FIG. 5 is a diagram for explaining normal work and deep digging work in the work vehicle shown in FIG. As shown in FIG. 4, the display system of the present embodiment includes a driver's cab 4, a head-mounted display 200 (display device), a main image pickup device ISM, a first complementary image pickup device IS1, and a second complementary image pickup device. It has an IS2, an image pickup device II, and a controller 50.

The head-mounted display 200 is mounted on the operator's head. The operator wearing the head-mounted display 200 sits in the driver's seat 4S in the driver's cab 4 and operates the work vehicle 100. Therefore, the head-mounted display 200 is located in the driver's cab 4.

The main image pickup device ISM is attached to the head-mounted display 200. The optical axis OA3 of the main image pickup apparatus ISM substantially coincides with the line-of-sight direction of the operator wearing the head-mounted display 200. In the case of FIG. 4, the optical axis OA3 substantially coincides with the horizon HL.

The optical axis OA1 of the first complementary image pickup apparatus IS1 faces forward and upward of the work vehicle 100 with the work vehicle 100 placed on a horizontal ground. Specifically, the optical axis OA1 is inclined so as to have an ascending gradient located upward as the distance from the first complementary image pickup apparatus IS1 increases. The optical axis OA1 of the first complementary image pickup apparatus IS1 is inclined at an angle α with respect to the horizon HL. When the work vehicle 100 is placed on a horizontal ground, the angle α is, for example, 20 °.

The optical axis OA2 of the second complementary image pickup apparatus IS2 faces forward and downward of the work vehicle 100 in a state where the work vehicle 100 is placed on a horizontal ground. Specifically, the optical axis OA2 is inclined so as to have a downward gradient located downward as the distance from the second complementary image pickup apparatus IS2 increases forward. The optical axis OA2 of the second complementary image pickup apparatus IS2 is inclined at an angle β with respect to the horizon HL. When the work vehicle 100 is placed on a horizontal ground, the angle β is, for example, 50 °. Further, the optical axis OA2 intersects with the optical axis OA1 in the vicinity of the eye point of the operator seated in the driver's seat 4S (near the main image pickup apparatus ISM), for example.

The imaging range VA2 of the second complementary imaging device IS2 is located below the imaging range VA1 of the first complementary imaging device IS1. The imaging range VA1 and the imaging range VA2 may partially overlap in front of the driver's cab 4.

When the imaging range VA1 and the imaging range VA2 overlap each other, a virtual line BL connecting the point where the imaging range VA1 and the imaging range VA2 start to overlap each other and the main imaging device ISM can be drawn in the side view.

The image pickup apparatus II is arranged inside the driver's cab 4. The image pickup apparatus II can take an image of the inside of the driver's cab 4, and can take an image of, for example, the head mounted display 200 and the vicinity of the head of the operator who wears the head mounted display 200. The image pickup apparatus II can detect the position of the head-mounted display 200 by taking an image of the head-mounted display 200.

Each of the main image pickup device ISM, the first complementary image pickup device IS1, the second complementary image pickup device IS2, and the image pickup device II is, for example, a compound-eye stereo camera. The stereo camera acquires point cloud data having position information in the front-back direction, the left-right direction, and the up-down direction of the image pickup target, and an image.

Here, the point cloud data is three-dimensional data having basic position information of orthogonal coordinates (X, Y, Z). By acquiring both the point cloud data and the image, for example, position information can be added to each pixel of the image. This makes it possible to create a three-dimensional CG (Computer Graphics) model from a single image.

Further, each of the main image pickup device ISM, the first complementary image pickup device IS1, the second complementary image pickup device IS2, and the image pickup device II is not limited to the stereo camera, and may be any as long as it can acquire an image having point cloud data. Each of the main image pickup device ISM, the first complementary image pickup device IS1, the second complementary image pickup device IS2, and the image pickup device II each has position information measured by LiDAR (Light Detection and Ranging) with respect to an image captured by a monocular camera, for example. May be added.

The work vehicle 100 has operating devices 21, 22, and 23 in the driver's cab 4. The operating devices 21, 22 and 23 have a pair of left and right operating levers 21, a forward / backward moving lever 22, and an operating pedal 23.

The pair of left and right operating levers 21 can operate the boom 6, the arm 7, and the bucket 8 to drive and the swivel body 3 to swivel with respect to the traveling body 5. The pair of left and right forward / backward levers 22 can operate the drive of the right crawler belt 5Cr and the left crawler belt 5Cr.

As shown in FIG. 5, the first complementary image pickup apparatus IS1 can take an image of a working machine 2 for normal work. Specifically, the imaging range VA1 of the first complementary imaging apparatus IS1 includes a working machine 2 for normal work. Further, the second complementary imaging device IS2 can image the working machine 2 for deep digging work. Specifically, the imaging range VA2 of the second complementary imaging apparatus IS2 includes a working machine 2 for deep digging work.

Whether the work machine 2 is in the state of normal work or the state of deep digging work may be determined based on, for example, the above-mentioned virtual line BL. Specifically, when the work machine 2 is located above the line BL, it is determined that the work machine 2 is performing normal work, and the work machine 2 is located below the line BL. In that case, it may be determined that the working machine 2 is carrying out the deep digging work.

Further, it may be determined whether the working machine 2 is in the state of normal work or the state of deep digging work by using a standard other than the line BL.

The hydraulic excavator 100 is equipped with a controller 50. Details of the controller 50 will be described later.

<Structure of head-mounted display>
Next, the configuration of the head-mounted display 200 shown in FIG. 4 will be described with reference to FIG.

FIG. 6 is a perspective view showing the configuration of the head-mounted display shown in FIG. As shown in FIG. 6, the head-mounted display 200 has a main body portion 101, an angle sensor 102, a mounting portion 103, and a main image pickup device ISM. The head-mounted display 200 is, for example, a transmissive display.

The main body 101 has a screen. This screen is arranged at a position that can be visually recognized by an operator wearing the head-mounted display 200. Images captured by the main image pickup device ISM, the first complementary image pickup device IS1, the second complementary image pickup device IS2, and the like are projected on this screen. An operator wearing the head-mounted display 200 can know the outside of the head-mounted display 200 by visually recognizing the image projected on the screen.

The angle sensor 102 is attached to, for example, the main body 101. The angle sensor 102 may be attached to a portion other than the main body portion 101, for example, the mounting portion 103. The angle sensor 102 detects the tilt angle of the head-mounted display 200.

The mounting portion 103 is a portion for mounting the head-mounted display 200 to the operator. The mounting portion 103 is, for example, a headband.

The main image pickup apparatus ISM is attached to the front surface of the main body 101. The main image pickup apparatus ISM images the side opposite to the main body 101. The main image pickup device ISM images the outside of the head-mounted display 200. The main image pickup apparatus ISM images the outside of the driver's cab 4 from the inside of the driver's cab 4.

<Structure of functional blocks of display system>
Next, the configuration of the functional block of the display system shown in FIG. 4 will be described with reference to FIG. 7.

FIG. 7 is a diagram showing an example of a functional block of the display system shown in FIG. As shown in FIG. 7, the controller 50 includes an image acquisition unit 50a, 50b, 50c, a detection information acquisition unit 50d, an image selection unit 50e, a viewpoint conversion unit 50f, an image superimposition unit 50g, and a head-mounted display. It has a control unit 50h, a position / angle calculation unit 50i, a work state determination unit 50j, and a storage unit 50k.

The image acquisition unit 50a acquires the data of the main image captured by the main image pickup device ISM. The image acquisition unit 50a outputs the acquired image data to the image superimposition unit 50g.

The image acquisition unit 50b acquires the data of the first complementary image captured by the first complementary imaging device IS1. The image acquisition unit 50b outputs the acquired data of the first complementary captured image to the image selection unit 50e.

The image acquisition unit 50c acquires the data of the second complementary image captured by the second complementary imager IS2. The image acquisition unit 50c outputs the acquired data of the second complementary captured image to the image selection unit 50e.

The detection information acquisition unit 50d acquires the position information and the angle information of the head-mounted display 200 detected by the position / angle detection sensor SE1. The position / angle detection sensor SE1 has, for example, an image pickup device II (FIG. 4) and an angle sensor 102 (FIG. 6).

The image pickup apparatus II acquires the data of the captured image of the head-mounted display 200 and outputs it as position information to the detection information acquisition unit 50d. The angle sensor 102 outputs the tilt angle of the head-mounted display 200 as angle information to the detection information acquisition unit 50d.

The detection information acquisition unit 50d outputs the acquired position information and angle information to the position / angle calculation unit 50i. The position / angle calculation unit 50i calculates the position and tilt angle of the head-mounted display 200 based on the acquired position information and angle information. Specifically, the position / angle calculation unit 50i calculates the tilt angle of the head-mounted display 200 with respect to the work vehicle 100 based on the tilt angle of the head-mounted display 200 detected by the angle sensor 102. Further, the position / angle calculation unit 50i calculates the position of the head-mounted display 200 based on the captured image data of the head-mounted display 200 imaged by the image pickup apparatus II.

The image pickup apparatus II acquires an image to which point cloud data having position information is added. Therefore, the position of the head-mounted display 200 can be calculated from the image captured by the image pickup apparatus II.

The position and tilt angle of the head-mounted display 200 calculated by the position / angle calculation unit 50i are output to each of the work state determination unit 50j and the viewpoint conversion unit 50f. The work state determination unit 50j determines whether the work vehicle 100 is performing normal work or deep digging work based on the position and tilt angle of the head-mounted display 200.

At the time of this determination, the work state determination unit 50j acquires a predetermined position and a predetermined tilt angle of the head-mounted display 200 from the storage unit 50k, and performs normal work or deep digging work based on the predetermined position and the predetermined tilt angle. Judge. This predetermined tilt angle is, for example, the tilt angle of the line BL in FIG. 4 or FIG.

Specifically, a state in which the head-mounted display 200 faces a predetermined tilt angle at a predetermined position or faces upward from the predetermined tilt angle is determined to be normal work. Further, a state in which the head-mounted display 200 faces downward from a predetermined tilt angle at a predetermined position is determined to be a deep digging operation. The signal indicating the determination result by the work state determination unit 50j is output to the image selection unit 50e.

The image selection unit 50e selects either the first complementary image or the second complementary image based on the working state of the work vehicle 100. Specifically, when it is determined that the work vehicle 100 is in the normal working state, the image selection unit 50e selects the first complementary image. Further, when it is determined that the work vehicle 100 is in the state of deep digging work, the image selection unit 50e selects the second complementary image. The image selection unit 50e outputs the data of the selected captured image to the viewpoint conversion unit 50f.

The viewpoint conversion unit 50f converts the captured image selected by the image selection unit 50e into an image of the operator's viewpoint sitting on the driver's seat 4S based on the position and tilt angle of the head-mounted display 200 acquired from the position / angle calculation unit 50i. Convert.

Each of the first complementary image pickup apparatus IS1 and the second complementary imaging apparatus IS2 acquires an image to which point cloud data having position information is added. Therefore, a three-dimensional CG model can be created from each of the first complementary image and the second complementary image. Therefore, both the first complementary image and the second complementary image can be converted into an image of the operator's viewpoint seated in the driver's seat 4S. The converted image converted into the image of the operator's viewpoint by the viewpoint conversion unit 50f is output to the image superimposing unit 50g.

The image superimposing unit 50g superimposes the main captured image acquired from the image acquisition unit 50a and the converted image acquired from the viewpoint conversion unit 50f. Here, the main captured image is an image captured by the main image pickup device ISM arranged in the driver's cab 4. Therefore, in addition to the scenery outside the driver's cab 4 (topography, work equipment 2, etc.), a shield such as a part of the driver's cab 4 (for example, pillars, floor, ceiling) is captured in the main captured image. On the other hand, the converted image is an image obtained by converting either the first complementary image or the second complementary image, and each of the first complementary image and the second complementary image is a landscape (topography,) outside the driver's cab 4. (Working machine 2 etc.) is imaged, and a shield such as a part of the driver's cab 4 is not imaged. Therefore, by superimposing the main captured image and the converted image, it is possible to generate a display image that penetrates the shield and complements the blind spot. The image superimposing unit 50g outputs the generated display image to the head-mounted display control unit 50h.

The head-mounted display control unit 50h controls the head-mounted display 200 so that the displayed image is displayed on the screen of the head-mounted display 200. As a result, the operator seated in the driver's seat 4S can visually recognize the display image projected on the screen of the head-mounted display 200 through which the obstruction is transmitted and the blind spot is complemented.

As described above, the controller 50 displays a display image complementing the blind spot on the screen of the head-mounted display 200 based on at least one complementary captured image of the first complementary captured image and the second complementary captured image.

Further, the controller 50 generates a display image complementing the blind spot by superimposing at least one of the first complementary image and the second complementary image and the main image on the screen of the head-mounted display 200. indicate.

Further, the controller 50 selects either the first complementary image or the second complementary image by the image selection unit 50e, so that the first complementary image and the second complementary image are superimposed as the image to be superimposed on the main image. To switch between.

Further, the controller 50 determines whether the work vehicle 100 is executing the normal work or the deep digging work by the work state determination unit 50j, and when it is determined that the work vehicle 100 is executing the normal work, the first The complementary image and the main image are superimposed, and when it is determined that the work vehicle 100 is executing the deep digging work, the second complementary image and the main image are superimposed.

The working state of the work vehicle 100 may be determined by the work state determination unit 50j based on the changeover signal of the changeover switch SW or the detection signal of the posture detection sensor SE2.

The changeover switch SW is operated by an operator boarded in the driver's cab 4. The changeover switch SW is configured to be able to switch at least between normal work and deep digging work. The changeover switch SW outputs a signal indicating the changeover state to the work state determination unit 50j. When the work state determination unit 50j acquires a signal indicating a normal state from the changeover switch SW, the work state determination unit 50j determines that the work vehicle 100 is in the normal state. Further, when the work state determination unit 50j acquires a signal indicating the deep moat state from the changeover switch SW, the work state determination unit 50j determines that the work vehicle 100 is in the deep moat state.

The posture detection sensor SE2 is a sensor for detecting the posture of the working machine 2. The attitude detection sensor SE2 is, for example, a stroke sensor, a potentiometer, an IMU (Inertial Measurement Unit), an image pickup device, an operation amount detection sensor by an operation lever 21, and the like.

When the stroke sensor is used as the posture detection sensor SE2, the stroke sensor is attached to each of the boom cylinder 10, the arm cylinder 11 and the bucket cylinder 12 shown in FIG. The stroke amount of each cylinder can be detected by the stroke sensor.

When a potentiometer is used as the attitude detection sensor SE2, the potentiometer is attached in the vicinity of each of the boom foot pin 13, the boom top pin 14 and the arm top pin 15 shown in FIG. Each potentiometer can detect each of the rotation angle of the boom 6 with respect to the main body 1, the rotation angle of the arm 7 with respect to the boom 6, and the rotation angle of the bucket 8 with respect to the arm 7.

When an IMU (Inertial Measurement Unit) is used as the attitude detection sensor SE2, the IMU is attached to each of the boom 6, arm 7, and bucket 8 shown in FIG. Each of the IMUs detects a triaxial angle (or angular velocity) and acceleration. The postures of the boom 6, the arm 7, and the bucket 8 can be detected by the angles (or angular velocities) and accelerations of the three axes detected by the IMU.

When an image pickup device is used as the attitude detection sensor SE2, the state of the boom 6, arm 7, and bucket 8 shown in FIG. 1 is imaged by the image pickup device. The postures of the boom 6, the arm 7, and the bucket 8 can be detected from the image pickup information captured by the image pickup device.

The operation amount detection sensor by the operation lever 21 is a pressure sensor that detects the pilot oil pressure (PPC pressure) when the operation lever 21 shown in FIGS. 4 and 10 is of the pilot hydraulic pressure system. Further, the operation amount detection sensor by the operation lever 21 is a potentiometer that detects the operation angle of the operation lever when the operation lever 21 is of an electric type.

The posture detection sensor SE2 outputs a signal indicating the posture of the work machine 2 to the work state determination unit 50j. The work state determination unit 50j determines whether the work vehicle 100 is in the normal state or the deep moat state based on the signal indicating the posture of the work machine 2.

<Display method>
Next, the display method of the work vehicle in the present embodiment will be described with reference to FIGS. 7 and 8.

FIG. 8 is a flow chart showing an example of a display method of a work vehicle according to an embodiment of the present disclosure. As shown in FIG. 8, information on the position and tilt angle of the head-mounted display 200 is acquired by the detection information acquisition unit 50d (FIG. 7) of the controller 50 (step S1). The main captured image is acquired by the image acquisition unit 50a (FIG. 7) of the controller 50 (step S2). The first complementary captured image is acquired by the image acquisition unit 50b (FIG. 7) of the controller 50 (step S3). The second complementary captured image is acquired by the image acquisition unit 50c (FIG. 7) of the controller 50 (step S4).

Based on the information on the position and tilt angle of the head-mounted display 200 acquired by the detection information acquisition unit 50d (FIG. 7), the position / angle calculation unit 50i of the controller 50 calculates the position and tilt angle of the head-mounted display 200. (Step S5). Based on the calculated position and tilt angle of the head-mounted display 200, the work state determination unit 50j (FIG. 7) of the controller 50 determines whether the work state of the work vehicle 100 is normal work or deep digging work (step S6). ..

When it is determined that the working state is the normal state, the first complementary captured image is selected by the image selection unit 50e of the controller 50 (step S7). When it is determined that the working state is the deep moat state, the second complementary captured image is selected by the image selection unit 50e (FIG. 7) of the controller 50 (step S8).

The complementary captured image selected from the first complementary captured image and the second complementary captured image is converted into an image of the operator's viewpoint seated in the driver's seat 4S by the viewpoint conversion unit 50f (FIG. 7) of the controller 50. (Step S9). This conversion is performed based on the position and tilt angle of the head-mounted display 200 calculated by the position / angle calculation unit 50i (FIG. 7).

The complementary image and the main image converted to the operator's viewpoint are superimposed on each other by the image superimposing unit 50g of the controller 50 (step S10). At this time, the complementary image and the main image converted to the operator's viewpoint so that the shield located in the direction from the inside of the driver's cab 4 toward the cutting edge 8a of the bucket 8 is transmitted and the blind spot is complemented. And are superimposed.

The superimposed display image is controlled by the head-mounted display control unit 50h of the controller 50 so as to be projected and displayed on the screen of the head-mounted display 200 (step S11). As a result, the operator wearing the head-mounted display 200 can visually recognize the superimposed display image through the screen of the head-mounted display 200.

[Image superimposition during Fukahori work]
Next, the superimposition of the complementary captured image converted to the operator's viewpoint and the main captured image during the deep-drilling work will be specifically described with reference to FIGS. 9 and 10.

FIG. 9 is a perspective view showing how the work vehicle is performing deep digging work. FIG. 10 is a diagram showing a state in which an image captured by the main image pickup apparatus and a second complementary image pickup image of the second complementary image pickup apparatus are superimposed during the work shown in FIG. In FIG. 10, the members (boom 6, arm 7, etc.) other than the bucket 8 of the working machine 2 are not shown for the sake of simplification.

During deep digging work as shown in FIG. 9, a deep vertical groove TR is formed on the ground on which the work vehicle 100 is placed. Therefore, the bucket 8 for excavating the bottom surface of the deep vertical groove TR is located considerably below the main body 1 of the work vehicle 100. Therefore, as shown in FIG. 10A, the operator seated in the driver's seat 4S in the driver's cab 4 may not be able to visually recognize the bucket 8 and the excavated surface due to the floor FR of the driver's cab 4. Therefore, in the main image pickup image CI2 captured from the operator's viewpoint by the main image pickup apparatus ISM (FIGS. 2 to 4), the bucket 8 and the excavated surface during the deep digging work may not be imaged.

Therefore, as shown in FIG. 10B, the second complementary image MI1 after the viewpoint conversion obtained by converting the second complementary image captured by the second complementary imager IS2 (FIGS. 1 and 4) into the operator viewpoint is obtained. Used. The second complementary image pickup apparatus IS2 is arranged outside the driver's cab 4. Therefore, the floor FR of the driver's cab 4 and the like are not captured in the second complementary captured image MI1 after the viewpoint conversion. Therefore, even if the bucket 8 during the deep digging work moves from the front side to the front side of the driver's cab 4 in the direction indicated by the arrow, it is possible to image the bucket 8 on the second complementary captured image MI1 after the viewpoint conversion.

The main captured image CI2 shown in FIG. 10 (A) and the second complementary captured image MI1 after the viewpoint conversion shown in FIG. 10 (B) are superimposed. As a result, as shown in FIG. 10C, a display image DI1 is generated in which a part of the floor FR of the driver's cab 4 is transmitted and the blind spot is complemented. This display image DI1 is projected on the screen of the head-mounted display 200. Through this screen, the operator wearing the head-mounted display can easily see the bucket 8 during the deep digging work through the floor portion FR.

In the display image DI1 shown in FIG. 10C, the operating devices 21, 22, 23, the display device DL, and the like may be displayed without being transmitted. As a result, the operator can easily operate the operation devices 21, 22, and 23, and can also visually recognize the display screen of the display device DL.

[Image superimposition during excavation work on slope]
Next, the superimposition of the complementary captured image converted to the operator's viewpoint and the main captured image during the excavation work of the slope will be specifically described with reference to FIGS. 11 and 12.

FIG. 11 is a side view showing how the work vehicle is excavating the slope SF. FIG. 12 is a diagram showing a state in which a main image captured by the main image pickup device and a first complementary image pickup image of the first complementary image pickup device are superimposed during the work shown in FIG.

During excavation work of the slope SF as shown in FIG. 11, the bucket 8 may be located above the upper surface of the driver's cab 4. Therefore, as shown in FIG. 12A, when the operator seated in the driver's seat 4S in the driver's cab 4 cannot visually recognize the bucket 8 and the excavated surface due to the front upper beam BT of the driver's cab 4, the roof portion RF, or the like. Occurs. Therefore, in the main image pickup image CI1a captured from the operator's viewpoint by the main image pickup apparatus ISM (FIGS. 4 and 6), the bucket 8 and the excavation surface during the excavation work of the slope SF may not be imaged.

Therefore, as shown in FIG. 12B, the first complementary image MI2 after the viewpoint conversion obtained by converting the first complementary image captured by the first complementary imager IS1 (FIGS. 1 and 4) into the operator viewpoint is obtained. Used. The first complementary image pickup apparatus IS1 is arranged outside the driver's cab 4. Therefore, the front upper beam BT, the roof portion RF, and the like of the driver's cab 4 are not captured in the first complementary captured image MI2 after the viewpoint conversion. Therefore, it is possible to capture the bucket 8 on the first complementary captured image MI2 after the viewpoint conversion even during the excavation work of the slope SF.

The main captured image CI1a shown in FIG. 12 (A) and the first complementary captured image MI2 shown in FIG. 12 (B) are superimposed. As a result, as shown in FIG. 12C, a display image DI2 is generated in which the front upper beam BT of the driver's cab 4, the roof portion RF, and the like are transmitted and the blind spot is complemented. This display image DI2 is projected on the screen of the head-mounted display 200. Through this screen, the operator wearing the head-mounted display can easily see the bucket 8 during the excavation work of the slope SF through the front upper beam BT, the roof portion RF, and the like.

In the display image DI2, not only the front upper beam BT and the roof portion RF but also the second front pillar FPR may be transmitted. Further, in the display image DI2 shown in FIG. 12C, the display device DL (FIG. 10) or the like may be displayed without being transmitted. This also enables the operator to visually recognize the display screen of the display device DL. Depending on the height of the field of view, the display screen of the monitor can be visually recognized.

[Image superimposition when loading a load onto a dump truck]
Next, the superposition of the complementary captured image converted to the operator's viewpoint and the main captured image at the time of loading the load onto the dump truck will be specifically described with reference to FIGS. 13 and 14.

FIG. 13 is a side view showing how the work vehicle loads the dump truck. FIG. 14 is a diagram showing a state in which an image captured by the main imaging device and an image captured by the first complementary imaging device are superimposed during the work shown in FIG.

At the time of loading the load into the dump truck 300 as shown in FIG. 13, the bucket 8 may be above the upper surface of the cab 4 and approach the front of the cab 4. Therefore, as shown in FIG. 14A, the operator seated in the driver's seat 4S in the driver's cab 4 may not be able to see the bucket 8 due to the second front pillar FPR of the driver's cab 4, the front upper beam BT, or the like. Occurs. Therefore, in the main image pickup image CI1b imaged from the operator's point of view using the main image pickup apparatus ISM (FIGS. 4 and 6), the bucket 8 during the loading operation may not be imaged.

Therefore, as shown in FIG. 14B, the first complementary image MI3 after the viewpoint conversion obtained by converting the first complementary image captured by the first complementary imager IS1 (FIGS. 1 and 4) into the operator viewpoint is obtained. Used. The first complementary image pickup apparatus IS1 is arranged outside the driver's cab 4. Therefore, the second front pillar FPR of the driver's cab 4, the front upper beam BT, and the like are not captured in the first complementary image MI3 after the viewpoint conversion. Therefore, it is possible to capture the bucket 8 on the first complementary captured image MI3 after the viewpoint conversion even during the loading operation.

The main captured image CI1b shown in FIG. 14 (A) and the first complementary captured image MI3 shown in FIG. 14 (B) are superimposed. As a result, as shown in FIG. 14C, a display image DI3 through which the second front pillar FPR of the driver's cab 4, the front upper beam BT, and the like are transmitted is generated. This display image DI3 is projected on the screen of the head-mounted display 200. Through this screen, the operator wearing the head-mounted display 200 can easily see the bucket 8 during the loading operation through the second front pillar FPR, the front upper beam BT, and the like.

Further, in the display image DI2 shown in FIG. 14C, the display device DL or the like may be displayed without being transmitted. This also enables the operator to visually recognize the display screen of the display device DL.

<Effect>
Next, the effect of this embodiment will be described.

According to the present embodiment, as shown in FIGS. 10, 12, and 14, a display in which the blind spot is complemented based on at least one of the first complementary image MI2, MI3, and the second complementary image MI1. Images DI1, DI2, and DI3 are displayed on the screen of the head-mounted display 200. Since the shield is transmitted through the display images DI1, DI2, and DI3 in which the blind spot is complemented, the operator can work while visually recognizing the work tool such as the bucket 8. As a result, accurate work can be performed, and it is not necessary to work in an unreasonable posture, and operator fatigue during work can be reduced.

Further, according to the present embodiment, as shown in FIGS. 10, 12 and 14, at least one of the first complementary image MI2, MI3 and the second complementary image MI1 and the main image CI2, CI1a , CI1b are superimposed to complement the blind spots of the main captured images CI2, CI1a, and CI1b. As a result, the operator can visually recognize the display images DI1, DI2, and DI3 in which the blind spot is complemented in the so-called video transmissive head-mounted display 200.

Further, according to the present embodiment, as shown in FIG. 7, the controller 50 switches between the first complementary image and the second complementary image as an image to be superimposed on the main image by the image selection unit 50e. By switching and superimposing the images in this way, it is possible to reduce the load of image processing as compared with the case where the images are integrated and superposed.

Further, according to the present embodiment, as shown in FIG. 7, the controller 50 determines whether the work machine 2 is executing the normal work or the deep digging work by the work state determination unit 50j. Then, when the controller 50 determines that the work machine 2 is executing the normal work, the controller 50 superimposes the first complementary image MI2 and the main image CI1a as shown in FIGS. 11 and 12. Further, when the controller 50 determines that the working machine 2 is executing the deep digging work, the controller 50 superimposes the second complementary captured image MI1 and the main captured image CI2 as shown in FIGS. 9 and 10. As a result, the displayed images DI1 and DI2 can be obtained by penetrating the shield and complementing the blind spot in both the normal work and the deep digging work.

Further, according to the present embodiment, as shown in FIG. 1, each of the first complementary image pickup apparatus IS1 and the second complementary imaging apparatus IS2 is arranged outside the cab 4. This makes it possible to obtain a complementary image in which the components of the driver's cab 4 are not captured.

Further, according to the present embodiment, as shown in FIG. 6, the display device is a head-mounted display 200 worn by the operator of the work vehicle 100. As a result, the operator can visually recognize the bucket 8 and the excavated surface as if he / she were directly visually recognizing through the display image complementing the blind spot.

<Modification example>
Next, a modification of the present embodiment will be described.

In the above embodiment, as shown in FIGS. 7 and 8, after either one of the first complementary image and the second complementary image is selected by the image selection unit 50e, the viewpoint is viewed by the viewpoint conversion unit 50f. Is converted.

However, as shown in FIGS. 15 and 16, after the viewpoints of the first complementary image and the second complementary image are converted by the viewpoint conversion units 50fa and 50fb, respectively, the first complementary image and the first complementary image after the viewpoint conversion Either one of the second complementary captured images may be selected by the image selection unit 50e.

In this case, as shown in FIG. 15, the image acquisition unit 50b outputs the acquired data of the first complementary captured image to the viewpoint conversion unit 50fa. Further, the image acquisition unit 50c outputs the acquired second complementary captured image data to the viewpoint conversion unit 50fb.

The viewpoint conversion unit 50fa converts the first complementary captured image into an image of the operator's viewpoint seated in the driver's seat 4S based on the position and tilt angle of the head-mounted display 200 acquired from the position / angle calculation unit 50i (step S9a). : FIG. 16). The viewpoint conversion unit 50fa outputs the data of the first complementary captured image after the viewpoint conversion to the image selection unit 50e.

Further, the viewpoint conversion unit 50fb converts the second complementary captured image into an image of the operator's viewpoint seated in the driver's seat 4S based on the position and tilt angle of the head-mounted display 200 acquired from the position / angle calculation unit 50i (step). S9b: FIG. 16). The viewpoint conversion unit 50fb outputs the data of the second complementary captured image after the viewpoint conversion to the image selection unit 50e.

The image selection unit 50e selects either the first complementary captured image or the second complementary captured image whose viewpoint has been changed based on the working state of the work vehicle 100. Specifically, when it is determined that the work vehicle 100 is in the normal working state, the image selection unit 50e selects the first complementary captured image whose viewpoint has been converted (step S7: FIG. 16). Further, when it is determined that the work vehicle 100 is in the state of deep digging work, the image selection unit 50e selects the second complementary captured image whose viewpoint has been changed (step S8: FIG. 16). The image selection unit 50e outputs the data of the selected captured image to the viewpoint conversion unit 50f.

Since the configurations of the functional blocks other than the above among the configurations of the functional blocks shown in FIG. 15 are almost the same as the configurations of the functional blocks shown in FIG. 7, the same elements are designated by the same reference numerals. Do not repeat the explanation.

Further, among the display methods shown in FIG. 16, steps other than the above are substantially the same as the steps shown in FIG. 8, so that the same steps are designated by the same reference numerals and the description thereof will not be repeated.

Further, in the above embodiment, the controller 50 selects either the first complementary image or the second complementary image by the image selection unit 50e and superimposes it on the main image to generate a display image. However, the controller 50 integrates the first complementary image and the second complementary image to generate one integrated image, and superimposes the integrated image and the main image to produce a display image transmitted through the shield. It may be generated.

Specifically, as shown in FIG. 17 (A), the viewpoint conversion unit 50fa (FIG. 15) of the controller 50 generates the first complementary captured image with the viewpoint converted. Further, as shown in FIG. 17 (B), the viewpoint conversion unit 50fb (FIG. 15) of the controller 50 generates a second complementary captured image whose viewpoint has been converted. After that, as shown in FIG. 17C, the controller 50 integrates the viewpoint-converted first complementary image and the viewpoint-converted second complementary image to generate one integrated image.

After that, the image superimposing unit 50g (FIG. 15) of the controller 50 superimposes the integrated image and the main captured image to generate a display image through which the shield in the main captured image is transmitted.

Further, in the above embodiment, the case where a part of the work vehicle 100 is transmitted in the display image has been described, but the portion transmitted in the display image may be a part of the terrain. Hereinafter, it will be described with reference to FIGS. 18 and 19 that a part of the terrain is transmitted in the displayed image.

18 and 19 are a front view and a side view showing a state of deep digging work. The VP in FIGS. 18 and 19 is an operator's viewpoint. Further, in FIG. 19, the illustration of the boom cylinder 10 is omitted for the sake of simplification of the figure. As shown in FIG. 18, in the case of deep digging work, a deep vertical groove TR is formed on the ground. In the deep digging work, the bottom surface BW of the vertical groove TR becomes the excavation surface. Therefore, it is preferable that the operator performs the deep digging work while visually recognizing the bottom surface BW of the vertical groove TR. However, when the vertical groove TR becomes deep, the wall surface SW1 of the vertical groove TR obstructs the operator's field of view. Therefore, the lower limit in the vertical groove TR that the operator can directly see is the wall surface SW2 of the vertical groove TR as shown by the arrow LS1 in FIG. Therefore, the operator seated in the driver's seat 4S cannot directly see the bottom surface BW which is the excavation surface.

Therefore, the second complementary imaging device IS2 images the inside of the vertical groove TR. The second complementary imaging device IS2 is preferably located in the region directly above the vertical groove TR in the front view of the work vehicle 100. Therefore, the second complementary image pickup apparatus IS2 can image the bottom surface BW of the vertical groove TR which is the excavation surface and the bucket 8 for excavating the bottom surface BW.

The second complementary image pickup apparatus IS2 captures an image to which point cloud data having position information is added as described above. Therefore, a three-dimensional CG model can be created from one image captured by the second complementary imaging device IS2. Therefore, the image captured by the second complementary imaging device IS2 can be converted into an image from the operator's viewpoint. Further, the wall surface SW1 can be identified based on the point cloud data, and the region R1 surrounded by the alternate long and short dash line in the figure including the wall surface SW1 can be deleted from the viewpoint-converted image. Therefore, by superimposing the second complementary captured image after the viewpoint conversion and the main captured image, it is possible to generate a display image in which the region R1 including the wall surface SW1 as the shielding portion is transmitted. By projecting this display image onto the screen of the head-mounted display 200, the operator can visually recognize the bottom surface BW and the bucket 8 which are the excavation surfaces as indicated by the arrows LS2 and LS3.

As shown in FIG. 19, in the case of deep digging work, when the vertical groove TR becomes deep, the front wall surface SW3 of the vertical groove TR obstructs the operator's field of view. Therefore, as shown by the arrow LS4 in FIG. 19, the operator may not be able to directly see the bucket 8.

Therefore, the second complementary imaging device IS2 images the inside of the vertical groove TR. The second complementary image pickup apparatus IS2 is arranged, for example, in front of the front surface of the driver's cab 4. Therefore, the second complementary image pickup apparatus IS2 can image the bottom surface BW of the vertical groove TR which is the excavation surface and the bucket 8 for excavating the bottom surface BW.

The second complementary image pickup apparatus IS2 captures an image to which point cloud data having position information is added as described above. Therefore, the image captured by the second complementary imaging device IS2 can be converted into an image from the operator's viewpoint. Further, the front wall surface SW3 can be identified based on the point cloud data, and the region R2 surrounded by the alternate long and short dash line in the figure including the front wall surface SW3 can be deleted from the viewpoint-converted image. Therefore, by superimposing the second complementary image taken after the viewpoint conversion and the main image taken, it is possible to generate a display image in which the region R2 including the front wall surface SW3 as a shielding portion is transmitted. By projecting this display image onto the screen of the head-mounted display 200, the operator can visually recognize the bottom surface BW and the bucket 8 to be the excavation surface as shown by the arrow LS5.

In the above, it has been described that a part of the work vehicle 100 or a part of the terrain is transmitted as a shield by superimposing the main image and the complementary image, but the transmitted shield is one of the work vehicles 100. Not limited to a part or part of the terrain. The transparent shield may be located between the operator and the work tool (for example, the bucket 8). In this case, an object located between the operator and the work tool is specified based on the point cloud data included in the main image and the complementary image, and the object is superimposed by superimposing the main image and the complementary image. It can be transmitted as a shield.

Further, when a part of the work vehicle 100 is to be transmitted, a portion to be transmitted may be set in advance, and the set portion may be stored in a storage unit 50k (FIG. 7, 15) or the like. In this case, the portion of the stored setting portion is cut out from the complementary imaging image captured by the first complementary imaging device IS1 or the second complementary imaging device IS2, and the image in the cut portion of the setting portion becomes the main captured image. It may be combined. In this way, a display image in which the shield is transmitted may be created at the set portion.

Further, the head-mounted display 200 shown in FIG. 6 may be an optical transmissive type. In this case, the main body 101 has a translucent screen. An operator wearing the head-mounted display 200 can visually recognize the outside of the head-mounted display 200 through a translucent screen. Further, an image captured by the first complementary imaging device IS1, the second complementary imaging device IS2, or the like is projected on the transflective screen. Therefore, the operator wearing the head-mounted display 200 can visually recognize the captured image projected on the translucent screen in addition to the external appearance transmitted through the translucent screen.

According to the optical transmissive head-mounted display 200, the operator can visually recognize the outside of the head-mounted display 200 through the semi-transmissive screen even when the power supply to the head-mounted display 200 is cut off. Therefore, even if the power supply to the head-mounted display 200 is cut off, the operator can perform the work using the work vehicle 100.

When the optical transmission type head-mounted display 200 is used, the main image pickup device ISM is omitted. In this case, at least one of the viewpoint-converted first complementary image and the second complementary image is cut out only in the shield portion. Then, only the image of the cut-out part of the shield is projected on the translucent screen. As a result, the image through which the shield is transmitted is projected onto the external appearance that the operator was visually recognizing through the translucent screen, so that the display image supplemented with the blind spot is projected on the translucent screen.

Further, the terrain of the displayed image through which the shield is transmitted and the target terrain may be projected on the screen of the head-mounted display 200 in a state of being superimposed on each other. Here, the target terrain is three-dimensional construction design information (design surface).

Further, the display image through which the obstruction is transmitted and the target terrain may be displayed with different chromaticities. For example, the part to be dug may be shown in red and the finished part may be shown in green with respect to the current terrain. Further, for example, the construction surface may be indicated by a wire frame, the scaffolding may be indicated by orange, and the construction surface may be indicated by green. Also, the part to be worked on today may be indicated by, for example, a yellow frame. This makes it easy to distinguish the display image through which the obstruction is transmitted from the target terrain.

In the above embodiment, the configuration in which the second complementary image pickup apparatus IS2 is attached to the driver's cab 4 has been described, but the second complementary imaging apparatus IS2 may be attached to the boom 6. In this case, the position of the second complementary imaging device IS2 is detected by, for example, an acceleration sensor, a potentiometer, or the like attached to the boom 6. Based on the detected position of the second complementary image pickup device IS2, the second complementary image pickup image captured by the second complementary image pickup apparatus IS2 is viewpoint-converted and superposed on the main image pickup image. As a result, it is possible to prevent the second complementary image pickup apparatus IS2 from interfering with the driving work machine.

Further, in the above embodiment, the configuration in which the first complementary imaging device IS1 and the second complementary imaging device IS2 are arranged vertically has been described, but other complementary imaging devices are the first complementary imaging device IS1 and the second complementary imaging device IS1 and the second complementary imager. It may be arranged in the left-right direction of the image pickup apparatus IS2. For example, a complementary image pickup device having an optical axis extending to the side of the work vehicle 100 or a complementary image pickup device having an optical axis extending to the rear of the work vehicle 100 may be added. This makes it possible to allow the engine cover 9, handrails, etc. of the work vehicle 100 to pass through as a shield.

Further, in the above embodiment, the case where the displayed image is projected on the screen of the head-mounted display 200 has been described, but the displayed image is displayed on a monitor or the like provided separately from the head-mounted display 200 in addition to the screen. You may.

Further, the controller 50 determines the tilt angle of the head-mounted display 200 with respect to the work vehicle 100, the line-of-sight direction of the operator detected by the line-of-sight detection sensor, the operation angle of the operation lever 21 operated by the operator, and the display switching operated by the operator. The first complementary image and the second complementary image are switched as an image to be superimposed on the main image based on one or more selected from the group consisting of the switching signal of the switch SW.

The tilt angle of the head-mounted display 200 can be detected by the angle sensor 102 attached to the head-mounted display 200. The operating angle of the operating lever 21 operated by the operator can be detected by an operating amount detection sensor (pressure sensor, potentiometer, etc.) of the operating lever 21.

The line-of-sight direction of the operator can be detected by the line-of-sight detection sensor attached to the head-mounted display 200. The line-of-sight detection sensor is, for example, a tracking sensor that tracks and detects the direction in which the pupils of the operator's left and right eyes are facing, and is, for example, an infrared camera. The line-of-sight direction of the operator is estimated as the direction in which a straight line connecting the position of the center of the eyeball and the position of the pupil, the iris, etc. in the image captured by the stereo camera faces, for example.

Further, whether or not to allow the wall surfaces SW1 and SW3 on the ground to pass through may be determined by AI (Artificial Intelligence).

The controller 50 shown in each of FIGS. 4, 7 and 15 may be mounted on the work vehicle 100 or may be arranged apart from the outside of the work vehicle 100. When the controller 50 is arranged apart from the outside of the work vehicle 100, the controller 50 may include a main image pickup device ISM, complementary image pickup devices IS1, IS2, a sensor SE1, a changeover switch SW, an attitude detection sensor SE2, a head-mounted display 200, and the like. May be wirelessly connected to. The controller 50 is, for example, a processor and may be a CPU (Central Processing Unit).

The storage unit 50k shown in each of FIGS. 5 and 13 may be built in the controller 50 or may be provided separately from the controller 50. The storage unit 50k is, for example, a memory.

It should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is shown by the scope of claims rather than the above description, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 main body, 2 work equipment, 3 swivel body, 4 driver's cab, 4S driver's seat, 5 traveling body, 5Cr cuff band, 5M traveling motor, 6 boom, 7 arm, 8 bucket, 8a cutting edge, 9 engine cover, 10 boom cylinder, 11 arm cylinder, 12 bucket cylinder, 13 boom foot pin, 14 boom top pin, 15 arm top pin, 21 operation lever, 22 forward / backward lever, 23 operation pedal, 50 controller, 50a, 50b, 50c image acquisition unit, 50d detection Information acquisition unit, 50e image selection unit, 50f, 50fa, 50fb viewpoint conversion unit, 50g image superimposition unit, 50h head mount display control unit, 50i angle calculation unit, 50j work state determination unit, 50k storage unit, 100 work vehicle, 101 Main body, 102 angle sensor, 103 mounting part, 200 head mount display, 300 dump truck, AP mounting part, BL line, BR1, BR2 bracket, BT front upper beam, BW bottom, CH1, CH2 holder, CI1a, CI1b, CI2 Main image, DI1, DI2, DI3 display image, DL display device, EX1 forward extension, EX2 downward extension, FPL 1st front pillar, FPR 2nd front pillar, FR floor, HL horizon, II imager , IS1 1st complementary imager, IS2 2nd complementary imager, ISM main imager, MI1 2nd complementary image, MI2, MI3 1st complementary image, OA1, OA2, OA3 optical axis, R1, R2 region, RF Roof, RX swivel axis, SE1 angle detection sensor, SE2 attitude detection sensor, SF slope, SW changeover switch, SW1, SW2 wall surface, SW3 front wall surface, TR vertical groove, VA1, VA2 image pickup range, VP operator viewpoint.

Claims (11)

Display device and
The first complementary imaging device and
A second complementary image pickup device that images below the image pickup range of the first complementary image pickup device, and a second complementary image pickup device.
A controller that displays a display image that complements the blind spot based on at least one of the first complementary image taken by the first complementary imager and the second complementary image taken by the second complementary imager. , A work vehicle display system. It is further equipped with a main image pickup device that captures the main image.
The work vehicle according to claim 1, wherein the controller generates the display image complementing the blind spot by superimposing the at least one complementary image and the main image, and displays the display image on the display device. Display system. The controller generates one integrated image in which the first complementary image and the second complementary image are integrated, and superimposes the integrated image and the main image to complement the blind spot. The work vehicle display system according to claim 2, which generates an image. The display system for a work vehicle according to claim 2, wherein the controller switches between the first complementary image and the second complementary image as an image superimposed on the main image. The controller detects switching between the first complementary image and the second complementary image as an image superimposed on the main image by the tilt angle of the display device with respect to the work vehicle and the line-of-sight detection sensor. The fourth aspect of claim 4, wherein the operation is performed based on one or more selected from the group consisting of the line-of-sight direction of the operator, the operation angle of the operation lever operated by the operator, and the changeover signal of the display changeover switch operated by the operator. Work vehicle display system. Equipped with more work equipment
The controller determines whether the working machine is performing the normal work or the deep digging work, and if it is determined that the working machine is performing the normal work, the first complementary image and the above are said. According to claim 4 or 5, the second complementary image and the main image are superimposed when it is determined that the work machine is performing the deep digging work by superimposing the main image. Display system for the described work vehicle. Further equipped with a driver's cab for operators to board
The display system for a work vehicle according to any one of claims 1 to 6, wherein the first complementary image pickup apparatus and the second complementary image pickup apparatus are arranged outside the driver's cab. The display system for a work vehicle according to any one of claims 1 to 7, wherein the display device is a head-mounted display worn by the operator of the work vehicle. With the work machine,
A driver's cab having a first front pillar and a second front pillar located closer to the working machine than the first front pillar.
A display device attached to the operator of the work vehicle and
Complementary imaging device and
A display system for a work vehicle, comprising: a controller for displaying a display image complemented by a blind spot by the second front pillar on the display device based on the complementary image taken by the complementary image pickup device. A driver's cab with a floor and
A display device attached to the operator of the work vehicle and
Complementary imaging device and
A display system for a work vehicle, comprising: a controller for displaying a display image complementing a blind spot by the floor portion of the driver's cab on the display device based on the complementary image pickup image of the complementary image pickup device. Display device and
The first complementary imaging device and
It is a display method of a work vehicle including a second complementary image pickup apparatus which images below the image pickup range of the first complementary image pickup apparatus.
The step of acquiring the first complementary image taken by the first complementary imager and
The step of acquiring the second complementary image taken by the second complementary imager, and
A method for displaying a work vehicle, comprising a step of displaying a display image in which a blind spot is complemented based on at least one of the first complementary image and the second complementary image on the display device.

Images