JP2022152012A - Construction machine - Google Patents

Abstract

To improve workability of an operator.SOLUTION: A construction machine 100 includes a main body that performs work, and a photographing device 200 that is provided outside an operation room 107 of the main body and photographs the surroundings to acquire a photographed image, and includes a display device 300 that performs a first display process for displaying the photographed image photographed by the photographing device in a predetermined image display area of a head mounted display. The image display area is between right and left operation lever images in a display visual field range, which is the visual field range of the operator on the head mounted display.SELECTED DRAWING: Figure 3

Description

The present invention relates to construction machinery, and more particularly to a technique for displaying an image of the surroundings of an operation room inside the operation room.

There is a construction machine called a deep foundation excavator that has an arm that deploys in multiple stages and is capable of excavating a deeper part from the ground surface than a normal hydraulic excavator. With this deep foundation excavator, if the hole to be excavated is deep, the bottom of the hole cannot be visually observed from the operation room, resulting in poor workability. Therefore, a mechanism for sliding the operation room forward and a floor window of the operation room are provided in order to secure a downward view. However, the field of view through the floor window of the operation room is limited by equipment such as pedals installed on the floor, and a wide field of view of the entire excavation hole cannot be obtained.

2. Description of the Related Art There is a system for displaying an image captured by a photographing device provided outside an operation room on a monitor device inside the operation room (see, for example, Patent Document 1). In the technique disclosed in Patent Document 1, "the head portion of the movable side cylinder of the telescopic cylinder is attached so as to move forward from the front position of the operating cabin, and the head portion is provided with a mounting angle adjusting means. A CCD camera is attached to the stage, and the transmission cable of the CCD camera is installed so as to bend in a substantially circular arc according to the operation of the movable side cylinder, is guided into the operating cabin, and is displayed on the monitor device provided in the operating cabin. Project the image data (summary excerpt)".

In addition, there is a technique for displaying an image captured by an imaging device provided outside the operation room of the construction machine on a head-mounted display (HMD) that covers the eyes of the operator of the construction machine (see, for example, Patent Document 2). . According to the technology disclosed in Patent Document 2, "the photographing unit includes a first photographing device that photographs an object from a first photographing direction. A first image display device is provided for displaying an image photographed by the photographing device toward an operation location, and the second image display device is arranged such that the inner product of the unit vector in the first photographing direction and the unit vector in the first display direction is a negative value. 1 imager and a first image display device are arranged (summary excerpt)”.

JP-A-11-36382 JP 2019-83370 A

In the technique disclosed in Patent Document 1, workability is improved to some extent by installing a monitor device. However, if the monitor device is installed on the front window and the left and right windows in the operation room, the surrounding field of vision is narrowed accordingly. Further, if the monitor device is installed near the feet of the operator, the shadow of the operator's knees prevents a large field of view from being obtained. In addition, when the monitor is placed at the feet, there is a risk of staining the monitor device. On the other hand, if the monitor device is placed above the knees of the operator, it interferes with the operator's smooth getting on and off and operation.

Further, in the technique disclosed in Patent Document 2, the display range is determined by the relative position with respect to the shooting direction, and the operator's visual field range is not taken into consideration. Therefore, when the display range overlaps with the operator's field of view, it becomes difficult to visually recognize the operating device and the like, which rather reduces workability.

SUMMARY OF THE INVENTION In view of the above, it is an object of the present invention to provide a technique for improving workability of an operator in a construction machine.

In order to achieve the above object, the construction machine of the present invention comprises a main body for executing work, and a photographing device provided outside an operation room of the main body for photographing the surroundings and obtaining a photographed image. and a display device for performing a first display process of displaying the captured image captured by the image capturing device in a predetermined image display area of a head mounted display, wherein the image display area is operated on the head mounted display. It is characterized by being between the left and right operation lever images in the display visual field range, which is the human visual field range.

Advantageous Effects of Invention According to the present invention, the workability of an operator is improved in a construction machine. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is an overall configuration diagram of a construction machine system according to a first embodiment; FIG. (a) is a top view of an operation room of the first embodiment, and (b) is an explanatory diagram for explaining an image example displayed on the display of the first embodiment. (a) is a photographing device of a first embodiment, and a functional block diagram of HMD, (b) is a hardware block diagram of the photographing device of a first embodiment. (a) and (b) are hardware block diagrams of HMD of 1st embodiment, respectively. (a) and (b) respectively show an example of the field-of-view image and the background image of the first embodiment, and (c) is an explanatory diagram for explaining a method of specifying the position of the image display area of the first embodiment. is. (a) and (b) are respectively explanatory diagrams for explaining a method of specifying the position of the video display area according to the first embodiment. 4A is an explanatory diagram for explaining an example of a display image of the first embodiment, and FIG. 4B is an explanatory diagram for explaining a scene seen by an operator through the display of the first embodiment; FIG. 4 is a flowchart of captured image display processing according to the first embodiment; It is an imaging device of 2nd embodiment, and a functional block diagram of HMD. (a) to (d) are explanatory diagrams for explaining non-display processing, size change processing, display position movement processing, and transparency processing, respectively, according to the second embodiment. 9 is a flowchart of captured image display processing according to the second embodiment. It is an imaging device of 3rd embodiment, and a functional block diagram of HMD. 10 is a flowchart of captured image display processing according to the third embodiment.

<<First Embodiment>>
A first embodiment of the present invention will be described.

First, a construction machine system 101, which is a visual enhancement system for a construction machine according to the present embodiment, will be described with reference to FIGS. 1 to 3. FIG. A construction machine system 101 of this embodiment includes a construction machine 100 , an imaging device (camera) 200 , and a display device 300 .

[Deep Foundation Excavator]
Hereinafter, in the present embodiment, a case where the construction machine 100 is a deep foundation excavator will be described as an example. As shown in FIG. 1 , the deep foundation excavator 100 includes a main body 102 for performing work and an operation room (cab) 107 .

The main body 102 includes a lower travel body 103 , an upper revolving body 105 provided on the lower travel body 103 so as to be able to revolve via a revolving shaft 104 , and a working device 106 provided on the upper revolving body 105 . An engine arranged in a machine room 108 and a counterweight 109 are mounted on the rear portion of the upper swing body 105 . The operation chamber 107 is mounted on the front lateral portion of the upper revolving body 105 .

A boom 110 and a boom cylinder 111 for rotating the boom 110 are rotatably supported on the upper swing body 105 . A tip portion of the boom cylinder 111 is rotatably connected to the boom 110 via a pin 111A. A multistage arm 112 is pivotally supported at the tip of the boom 110 so as to be rotated by an arm cylinder 113 . A clamshell bucket 114 , which is a tip attachment, is rotatably supported on the multistage arm 112 via a link portion 115 .

Here, FIG. 2A is a top view of the operation chamber 107. FIG. As shown in FIGS. 1 and 2(a), the operation room 107 includes a seat 118 as a seat for the operator M, a seat belt 124 for protecting the operator M, and the main body 102 arranged on the left and right sides of the seat 118. , a lock lever 120 for cutting input transmission from the operation lever 119 to the body 102 , and an operation room front window 121 for viewing the front of the body 102 .

[Shooting device]
The photographing device 200 is a camera for photographing the periphery (surroundings) of the main body, as shown in FIG. The imaging device 200 is provided outside the operation room 107 of the main body 102 . The imaging device 200 is arranged at a position where at least part of the working device 106 is within the imaging range. The photographing device 200 is supported by a main body 102 including a boom 110, a multi-stage arm 112, and the like, or a support section (not shown) arranged around the main body 102. As shown in FIG. In FIG. 1, the photographing device 200 photographs in the direction of arrow A, for example. Photographed data (captured video) photographed by the photographing device 200 is transmitted to the display device 300 . Details of the imaging device 200 will be described later.

[Display device]
The display device 300 displays the image captured by the image capturing device 200 to the operator M. FIG. In this embodiment, the display device 300 will be described as a head mounted display (HMD) worn on the head by the operator M so as to cover both eyes. Also, in this embodiment, the HMD 300 will be described as an example of an optical see-through type including a transmissive display as the display 318 (FIG. 2B). That is, the operator M can see the photographed image displayed on the display 318 and the real scenery (background real image) in the background thereof through the display 318 of the HMD 300 .

FIG. 2B shows an example of an image displayed on the display 318 of the HMD 300. As shown in FIG. As described above, operator M views captured image 370 and background real image 361 through display 318 . HMD 300 displays captured image 370 in predetermined image display area 350 of display area 360 on display 318 . HMD 300 displays nothing in other areas of display area 360 of display 318 . Therefore, the operator M can see the actual scenery of the background.

As shown in FIG. 2B, for example, the image display area 350 is near the image of the leg of the operator M in the background real image 361 seen by the operator M through the display 318, and the operation An area that does not interfere with the work of the person M, for example, an area in which at least a part of the image of the operating lever 119 (operating lever image 119i) can be visually recognized is set. Specifically, as an example, as shown in this figure, the image display area 350 includes the operation lever images 119i on both sides, the lower image of the operation room front window 121 (floor front end image 131i), and the floor of the operation room 107. The image of the rear end or the front end of the seat 118 (floor rear end image 132i) is set in the inner region of the region surrounded by.

Details of the imaging device 200 and the display device 300 (HMD 300) will be described below. FIG. 3A is a functional block diagram of the imaging device 200 and the display device 300 of this embodiment. 3B is a hardware configuration diagram of the imaging device 200, and FIG. 4A is a hardware configuration diagram of the HMD 300, respectively.

[Shooting device]
As shown in FIG. 3( a ), the imaging device 200 includes a captured image generation unit 221 and an image transmission unit 222 . Further, as shown in FIG. 3B, the photographing apparatus 200 includes an arithmetic device 211 such as a CPU, a RAM 212, a storage device 213 such as a ROM or a flash memory, a communication interface (I/F) 214, a CCD and an image sensor 215 such as.

The captured image generation unit 221 acquires signals from the image sensor 215 at predetermined time intervals and generates a captured image 370 . Image transmission unit 222 transmits captured image 370 generated by captured image generation unit 221 to the outside via communication I/F 214 . In this embodiment, it is transmitted to the display device (HMD) 300 . Transmission may be wired or wireless.

Note that the captured image generating unit 221 and the image transmitting unit 222 are realized by the arithmetic device 211 loading a program stored in advance in the storage device 213 into the RAM 212 and executing the program.

[HMD]
Next, the structure of HMD300 of this embodiment is demonstrated. As shown in FIG. 3( a ), the HMD 300 includes an image receiving section 321 , a relative position calculating section 322 , a display image generating section 323 , a display section 324 and a data storage section 331 . Further, as shown in FIG. 4A, the HMD 300 of the present embodiment includes an arithmetic device 311 such as a CPU, a RAM 312, a storage device 313 such as a ROM or a flash memory, a communication I/F 314, an input/output I/F. A /F 315 , a projector 316 , a lens 317 , a display 318 and a line-of-sight camera 319 are provided.

Projector 316 outputs the image generated by display image generation unit 323 . Also, the lens 317 enlarges or reduces the image output from the projector 316 . A display 318 is a transmissive display on which an image that has passed through a lens 317 is projected, and is composed of, for example, a half mirror.

The display 318 displays an image and can superimpose the image on the real surrounding environment (background real image 361) within the field of view of the operator M due to its transparency. A half mirror that constitutes the display 318 is arranged in front of the pupil of the operator M wearing the HMD 300 and has both transmissivity and reflectivity. As a result, the operator M can visually recognize the image projected on the display 318 and the actual scenery in front at the same time.

The line-of-sight camera 319 is a measurement device that acquires an image for measuring the operator M's viewpoint position and line-of-sight direction. The line-of-sight camera 319 is mounted on the main body of the HMD 300, and changes its position and optical axis direction according to the movement of the operator M's head. In this embodiment, the line-of-sight camera 319 acquires an image of the operator M's visual field range (visual field image).

Note that the HMD 300 may include a video see-through display 318 . A hardware configuration diagram of the HMD 300 in this case is shown in FIG. As shown in this figure, the video see-through type display 318 is, for example, a liquid crystal monitor or an organic EL monitor.

In addition, the HMD 300 may include a gyro sensor, an acceleration sensor, a distance sensor, a GPS antenna and receiver, instruction buttons, and the like.

Video reception unit 321 receives video from the outside via communication I/F 314 . In this embodiment, the captured image 370 is received from the imaging device 200 .

The relative position calculator 322 performs video display area identification processing. The image display area specifying process is a process of calculating the position of the image display area 350 within the display area of the display 318 of the HMD 300 . The image display area 350 is an area where the captured image 370 is displayed (superimposed) on the display 318 of the HMD 300, as described above. Also, the calculated value is positional information specifying the video display area 350 within the display area of the display 318 . For example, in this embodiment, it is the pixel position of the display image displayed on the display 318 .

In this embodiment, a predetermined area within the visual field range (display visual field range) of the operator M on the display 318 is defined as the video display area 350 . For this reason, the relative position calculation unit 322 first acquires an image (view image 380) of the visual field range in the line-of-sight direction of the operator M using the line-of-sight camera 319 . An example of the acquired field-of-view image 380 is shown in FIG. 5(a).

Note that the visual field image 380 is an image displayed as a background image when the video see-through type display 318 is used, as shown in FIG. 5(b). The field-of-view image 380 acquired here may be a video of only the operator M's field-of-view range, or an image of a size range equivalent to that of the display 318 (same size as the display area 360) including the display field-of-view range. There may be.

The relative position calculator 322 analyzes the acquired latest image of the field-of-view image 380 and identifies the position of the video display area 350 within the field-of-view image 380 . In this embodiment, the image display area 350 is a rectangular area surrounded by the floor front edge image 131i, the floor rear edge image 132i, and the left and right operation lever images 119i of the operation room 107 in the visual field image 380, as described above. be. For example, as shown in FIG. 5C, the pixel positions (field-of-view image coordinates) on the field-of-view image 380 of the four corners 351, 352, 353, and 354 of this rectangle are specified.

In specifying the pixel positions of the four corners 351, 352, 353, and 354 of the image display area 350, the relative position calculation unit 322 analyzes the field of view image 380, and determines the floor front edge image 131i, floor rear edge image 132i, and floor edge image 132i in the field of view image 380. The pixel positions of the left and right operation lever images 119i are specified. Here, for example, pattern matching or the like is performed for identification. That is, the relative position calculator 322 scans the field-of-view image 380, extracts each shape, and specifies the pixel position on each field-of-view image. Also, the analysis is performed at predetermined time intervals.

Shape data (pattern matching data) used for pattern matching is stored in the data storage unit 331 in advance. In this embodiment, as pattern matching data, the shapes of the floor front end image 131i, the floor rear end image 132i, and the left and right operation lever images 119i for each model are stored in advance. For example, a large number of images may be stored with different sizes, tilts, etc., or the field-of-view image 380 may be deformed during matching.

The relative position calculator 322 outputs the pixel positions of the four corners 351 , 352 , 353 , and 354 of the identified video display area 350 to the display video generator 323 .

Note that the image display area 350 does not necessarily have to be near the center of the field-of-view image 380 . For example, they may be unevenly distributed as shown in FIG. 6(a), or may have a predetermined angle. In addition, as shown in FIG. 6B, the relative position calculation unit 322 may not be able to obtain the pixel positions of all the four corners 351, 352, 353, and 354, such as when the line of sight of the operator M is not at the feet. . In such a case, the pixel position is output to the display image generation unit 323, but the display image generation unit 323 does not generate the display image for the range protruding from the field-of-view image 380. FIG.

Note that the video display area specifying method by the relative position calculation unit 322 is not limited to the above. Various conventional techniques can be used. For example, a marker or the like is attached at a predetermined position in the operation room 107 , and the coordinate values of the four corners 351 , 352 , 353 , 354 of the image display area 350 are calculated using the marker as a clue, and the pixels of the display image displayed on the display 318 are calculated. You can convert to position. In addition, the relative positions of the HMD 300 and various parts in the operation room 107 are calculated using various sensor signals such as a gyro sensor and an acceleration sensor provided in the HMD 300 and sensors for detecting the position of the HMD 300 provided in the cab, A video display area may be specified.

The display image generation unit 323 performs display image generation processing for generating a display image to be displayed on the display 318 . In the present embodiment, a display image is generated in which the captured image 370 is displayed in the area determined by the pixel positions of the four corners 351 , 352 , 353 , and 354 of the image display area 350 output from the relative position calculator 322 .

Here, FIG. 7A shows an example of a display image 390 generated by the display image generation unit 323. As shown in FIG. In this embodiment, as described above, a transmissive display is used, and the area other than the image display area 350 displaying the captured image 370 is transparent (transmittance is 100%). Thereby, the operator M can see the actual scenery (background real image 361) from outside the video display area 350. FIG. In addition, in FIG. 7A, the background real image 361 is indicated by a thin line to indicate the position.

Note that even when the relative position calculation unit 322 cannot calculate the pixel positions of all the four corners 351, 352, 353, and 354, the pixel positions are output to the display image generation unit 323, but the pixel positions protrude from the visual field image 380. As for the range, the display image generation unit 323 does not generate a display image.

In addition, instead of generating a display image with a transparent background, the display image generation unit 323 notifies the display unit 324 of information on the size of the captured image 370 and information on the display position determined by the image display area 350. may

The display unit 324 causes the display 318 to display the display image 390 generated by the display image generation unit 323 . In this embodiment, the input/output I/F 315 outputs to the projector 316 . As a result, the operator M can see a captured image 370 displayed in an image display area 350 on the display 318 via the display 318 of the HMD 300, and a background real image 361 in other areas, as shown in FIG. A display image 390 is viewable through the display 318 .

The data storage unit 331 stores data used for processing by each unit of the present embodiment, data obtained by the processing, and the like. For example, as described above, pattern matching data and the like are stored. The data storage unit 331 is constructed in the storage device 313 or the like.

[Processing flow]
The flow of the captured image display processing of this embodiment will be described below. FIG. 8 is a processing flow of the captured image display processing of this embodiment. This process is executed each time the video receiving unit 321 receives the captured video 370 .

When the image receiving unit 321 receives the captured image 370, the relative position calculating unit 322 performs image display area specifying processing (step S1101). The display area identification process analyzes the latest visual field image 380 acquired by the line-of-sight camera 319 at that time, and calculates the pixel positions of the four corners 351, 352, 353, and 354 of the image display area 350 on the visual field image 380. is.

Here, the relative position calculator 322 determines whether or not the image display area 350 has been identified (step S1102). Specifically, when the pixel positions of all four corners 351, 352, 353, and 354 can be calculated, it is determined that they have been specified. Otherwise, it is determined that the identification is impossible. Then, if the identification is not possible, the process is terminated as it is.

On the other hand, if it can be specified (S1102; Yes), the pixel positions of the four corners 351, 352, 353, and 354 are transferred to the display image generation unit 323. FIG. The received display image generation unit 323 performs display image generation processing (step S1103). The display image generation process is a process of generating the display image 390 so that the captured image 370 received by the image receiving unit 321 is displayed in the area specified by the pixel positions of the four corners 351 , 352 , 353 , and 354 .

When the display image generation unit 323 generates the display image 390, the display unit 324 displays the generated display image 390 (step S1104), and ends the process.

As a result, when the operator M wears the HMD 300 and the line-of-sight direction is the foot direction in the operation room 107, the captured image 370 captured by the image capturing device 200 is displayed in the predetermined image display area 350. be.

As described above, the construction machine system 101 of the present embodiment is provided outside the main body 102 of the construction machine (the excavator 100) that performs work and the operation room 107 of the main body 102, and photographs the surroundings. a photographing device 200 for obtaining a photographed image 370, and a display device 300 for performing first display processing for displaying the photographed image 370 in a predetermined image display area 350 of a display 318. The image display area 350 is surrounded by the operation lever image 119i, which is a background real image 361 visible through the display 318, within the display visual field range, which is the visual field range of the operator M on the display 318. It is characterized by being an area that does not interfere with the work of M.

As described above, according to the present embodiment, the image captured by the image capturing device 200 is displayed in the image display area 350 so that at least a part of the operation lever 119 can be visually recognized within the field of view of the operator M through the HMD 300 . A video 370 is displayed. That is, according to this embodiment, the display of the captured image 370 does not interfere with the lever operation. Therefore, the operator M can view the captured image 370 in the real space viewed through the HMD 300 without obstructing the view of the space necessary for the operation. As a result, the operator M can visually recognize the necessary image outside the operation room 107 without interfering with the operation, thereby improving workability.

An image display area 350 for displaying the photographed image 370 is set in an area inside the area surrounded by the operation lever image 119i and the floor front edge image 131i and the floor rear edge image 132i of the operation room 107. FIG. Therefore, according to the present embodiment, the photographed image 370 is displayed in a space above the legs of the operator M, which is a space that does not affect the surrounding field of vision and lever operation unique to the operation room 107 of the deep foundation excavator 100 . Also, this space is similar to the location of a conventional control room floor window. Therefore, the operator M can obtain a wide field of view outside the operation room without feeling discomfort and being prevented from operating the deep-foundation excavator 100 . As a result, the operator M can safely operate the deep-foundation excavator 100, improving workability.

In addition, since the photographed image 370 is displayed only when the line of sight of the operator M is at the feet, the photographed image 370 can be visually recognized in a manner similar to the case of visually recognizing the photographed image 370 from the floor window of the operation room. , there is less sense of incongruity. Furthermore, when the line-of-sight direction is different, such as when getting on and off the vehicle, the captured image 370 is not displayed. Therefore, you can get on and off safely.

<Modification 1>
It should be noted that in the above embodiment, display is not performed when all the pixel positions of the four corners 351, 352, 353, and 354 cannot be obtained, but the present invention is not limited to this. For example, if at least one image is obtained, the captured image 370 may be displayed. That is, when even part of the image display area 350 enters the visual field range (within the visual field image 380), the captured image 370 is displayed. For example, it is a case like FIG.6(b).

In this case, the relative position calculator 322 outputs the calculated pixel positions among the pixel positions of the four corners 351 , 352 , 353 , and 354 to the display image generator 323 . The display image generation unit 323 identifies the image display area 350 in the field-of-view image 380 from the received pixel positions, and displays the captured image 370 in the area.

<<Second Embodiment>>
Next, a second embodiment of the invention will be described. In this embodiment, it is further determined whether or not the operator M is seated, and the display of the captured image 370 is controlled.

Hereinafter, the present embodiment will be described with a focus on the configuration different from that of the first embodiment. Since the hardware configurations of the imaging device 200 and the HMD 300 and the functional blocks of the imaging device 200 are the same as those in the first embodiment, description thereof will be omitted here.

In this embodiment, whether or not the operator M is seated is detected by a seating sensor installed on the seat. Therefore, the deep foundation excavator 100 of the present embodiment includes a seating sensor 131 (see FIG. 9). The seating sensor 131 is arranged, for example, on the back of the seat. It may also be arranged in the lower part of the backrest of the seat.

The seating sensor 131 outputs a signal to the HMD 300 when it detects that the operator M is seated.

FIG. 9 is a functional block of the HMD 300 of this embodiment. The HMD 300 of this embodiment further includes a seating determination section 325 in addition to the configuration of the first embodiment.

When receiving a signal from the seating sensor 131 , the seating determination unit 325 outputs a seating signal indicating that the operator M has sat down to the display image generation unit 323 .

The display image generation unit 323 of the present embodiment performs different display image generation processing depending on whether the seating signal is received or not. When the seat signal is received, the seat display image generation processing is performed. The seated display image generation processing is processing for generating the display image 390 so that the captured image 370 is displayed in the image display area 350, as in the first embodiment.

When the display image generation unit 323 generates the display image 390 by the sitting display image generation processing, the display unit 324 displays the display image 390 on the display 318 as in the first embodiment. The seated display image generation processing by the display image generation unit 323 and the display processing by the display unit 324 are collectively referred to as first display processing.

On the other hand, when the seating signal is not received, the display image generation unit 323 performs display image generation processing when the user is not seated. The non-sitting display image generation processing includes, for example, non-display processing for hiding the captured image 370, size change processing for reducing the size of the image display area 350 and displaying the captured image 370, and display of the image display area 350 to a predetermined size. It is either display position movement processing for moving to a position and displaying the shot image 370 or transparency processing for displaying the shot image 370 in a transparent manner.

When the display image generation unit 323 generates the display image 390 by the non-sitting display image generation processing, the display unit 324 displays the display image 390 on the display 318 as in the first embodiment. Note that the non-sitting display image generation processing by the display image generation unit 323 and the display processing by the display unit 324 are collectively referred to as second display processing.

The non-display process is a process of not displaying the captured image 370 as shown in the right diagram of FIG. 10(a). That is, when it is designated to perform non-display processing, the display image generation unit 323 generates a display image 390 that is transparent on the entire screen. Then, the display unit 324 displays the display image 390 on the display 318 . Note that the display image 390 itself does not have to be generated at the timing when the non-display process is specified. In this case, the display unit 324 does not display itself. The left diagram of FIG. 10A is a display example when the first display processing is performed. The same applies to FIGS. 10(b) to 10(d).

The size change process is a process of changing the display size of the captured image 370 . In this embodiment, the display size is reduced. As shown in the right diagram of FIG. 10B, in this embodiment, the image display area 350 is reduced. A reduction ratio when a seating signal is received is determined in advance and stored in the data storage unit 331 . The display image generation unit 323 generates a display image such that the captured image 370 is displayed within the image display area 350 reduced at a predetermined reduction ratio. At this time, the center pixel position of the image display area 350 may be the same before and after the reduction.

The display position moving process is a process of moving the display position of the captured image 370 to a predetermined position. As shown in the right diagram of FIG. 10(c), the position of the image display area 350 within the display area of the display 318 is determined by the information calculated by the relative position calculation unit 322 (initial position). The destination position is determined in advance and stored in the data storage unit 331 . The destination may be outside the field of view image 380, for example.

As shown in the right diagram of FIG. 10( d ), the transmissive processing of the captured image is a process of changing the transmittance of the captured image to obtain a semi-transparent image 371 . For example, when the captured image 370 has transmission information (α value), the transmittance of the captured image 370 is changed by changing the α value.

Note that the size change process, the display position movement process, and the transparency process may be combined arbitrarily. For example, the photographed image 370 to be displayed is reduced in size to increase transparency, and the display position is moved.

[Processing flow]
The flow of the captured image display processing of this embodiment will be described below. FIG. 11 is a processing flow of captured image display processing according to the present embodiment. This process is executed each time the video reception unit 321 receives a video. Hereinafter, the same reference numerals are given to the same processes as in the first embodiment.

When the image receiving unit 321 receives the captured image, the relative position calculating unit 322 performs image display area specifying processing, as in the first embodiment (step S1101).

Here, the relative position calculator 322 determines whether or not the image display area 350 has been identified (step S1102), as in the first embodiment. If the identification is not possible, the process is terminated as it is.

If it can be specified (S1102; Yes), the pixel positions of the four corners 351, 352, 353, and 354 are transferred to the display image generation unit 323. FIG.

The display image generation unit 323 that has received the pixel positions from the relative position calculation unit 322 first determines whether or not the operator M is seated (step S2101). In the present embodiment, when receiving a seating signal from the seating sensor 131 , the seating determination unit 325 outputs the seating signal to the display image generation unit 323 . Therefore, the display image generator 323 determines whether or not the seating signal has been received within the predetermined period.

When it is determined that the user is seated (S2101; Yes), the display image generation unit 323 performs a display image generation process when seated (step S2102). The sitting-time display image generation processing is the same as the display image generation processing of the first embodiment.

Then, when the display image generation unit 323 generates the display image 390, the display unit 324 displays the generated display image 390 (step S1104), and ends the processing.

On the other hand, if it is not determined that the person is seated (S2101; No), the display image generation unit 323 performs non-sitting display image generation processing (step S2103), and proceeds to step S1104.

Thus, the construction machine system 101 of this embodiment has the same configuration as that of the first embodiment. Therefore, according to this embodiment, the same effects as those of the first embodiment can be obtained.

Furthermore, according to this embodiment, the display mode is changed depending on whether or not the operator M is seated. Specifically, when the operator M is not seated, for example, when getting in and out of the vehicle, the captured image 370 is not displayed or displayed so as not to be an obstacle. As a result, it is possible to prevent a situation that obstructs the operator's M field of vision, thereby improving safety and further improving workability.

<Modification 2>
In the above embodiment, whether or not the operator M is seated is determined based on the signal from the seating sensor 131 . However, the seating determination is not limited to this.

For example, the determination may be based on whether the seat belt 124 is worn. In this case, a signal output when the seat belt 124 is worn, such as a sensor provided on the buckle of the seat belt 124, is used. That is, when receiving the signal indicating that the seat belt is fastened, the seating determination unit 325 determines that the user is seated, and outputs a seating signal to the display image generation unit 323 .

Alternatively, the position of the HMD 300 may be used for determination. In this case, for example, the relative position calculator 322 identifies the position of the HMD 300 within the operation room 107 by analyzing the visual field image 380 of the line-of-sight camera 319 . Here, in particular, the position of the HMD 300 in the height direction is specified. Based on the specified position in the height direction, it is determined whether or not the operator M wearing the HMD 300 is in a posture equivalent to sitting. For the determination, for example, a predetermined threshold range is provided, and if the calculated position in the height direction is within the threshold range, it is determined that the person is seated.

Furthermore, the seating determination may be made using any combination of these techniques.

<Modification 3>
It should be noted that whether the person is seated or not may be determined prior to the image display area specifying process. In particular, if the non-sitting display process is a non-display process, and if it is determined that the user is not seated, the process ends. As a result, it is possible to reduce the processing amount when the user is not seated.

<<Third Embodiment>>
Next, a third embodiment of the invention will be described. In this embodiment, the state of the lock lever 120 is used to determine whether or not to display the image.

Hereinafter, the present embodiment will be described with a focus on the configuration different from that of the first embodiment. Since the hardware configurations of the imaging device 200 and the HMD 300 and the functional blocks of the imaging device 200 are the same as those in the first embodiment, description thereof will be omitted here.

FIG. 12 is a functional block of the HMD 300 of this embodiment. The HMD 300 of this embodiment further includes a lock lever state determination section 326 in addition to the configuration of the first embodiment. When the lock lever 120 is set, that is, when the operation lever 119 and the like are locked, a signal is output.

Upon receiving the signal from the lock lever, lock lever state determination section 326 outputs a lock signal indicating that operation lever 119 and the like are in the locked state to display image generation section 323 .

The display image generation unit 323 of the present embodiment performs different display image generation processing depending on whether the lock signal is received or not. When the lock signal is not received, image display processing is performed when unlocked. As in the first embodiment, the unlocked image display process is a process of generating the display image 390 so that the captured image 370 is displayed in the image display area 350 . As in the second embodiment, the unlocked-time video generation processing by the display video generation unit 323 and the display processing by the display unit 324 are collectively referred to as first display processing.

On the other hand, when the lock signal is not received, the image generating process at lock is performed. Like the non-sitting process of the second embodiment, the lock process is one of non-display process, size change process, display position movement process, and transparency process. As in the second embodiment, the lock-time display image generation processing by the display image generation unit 323 and the display processing by the display unit 324 are collectively referred to as second display processing.

[Processing flow]
The flow of the captured image display processing of this embodiment will be described below. FIG. 13 is a processing flow of captured image display processing according to the present embodiment. This process is executed each time the video reception unit 321 receives a video. Hereinafter, the same reference numerals are given to the same processes as in the first embodiment.

When the image receiving unit 321 receives the captured image, the relative position calculating unit 322 performs image display area specifying processing, as in the first embodiment (step S1101).

Here, the relative position calculator 322 determines whether or not the image display area 350 has been identified (step S1102), as in the first embodiment. If the identification is not possible, the process is terminated as it is.

If it can be specified (S1102; Yes), the pixel positions of the four corners 351, 352, 353, and 354 are transferred to the display image generation unit 323. FIG.

The display image generation unit 323 that has received the pixel positions from the relative position calculation unit 322 first determines whether or not the lock lever 120 is in the locked state (step S3101). In this embodiment, upon receiving a signal from the lock lever 120 , the lock lever state determination section 326 outputs a lock signal to the display image generation section 323 . Therefore, the display image generator 323 determines whether or not the lock signal has been received within the predetermined period.

If it is determined to be in the unlocked state (S3101; Yes), the display image generation unit 323 performs unlocked display image generation processing (step S3102). The unlocked display image generation processing is the same as the display image generation processing of the first embodiment.

Then, when the display image generation unit 323 generates the display image 390, the display unit 324 displays the generated display image 390 (step S1104), and ends the process.

On the other hand, if it is determined to be in the locked state (S2101; No), the display image generation unit 323 performs lock display image generation processing (step S3103), and proceeds to step S1104.

Thus, the construction machine system 101 of this embodiment has the same configuration as that of the first embodiment. Therefore, according to this embodiment, the same effects as those of the first embodiment can be obtained.

Furthermore, according to this embodiment, the displayed image is changed according to the state of the lock lever 120 . Specifically, in the locked state, the captured image 370 is not displayed or displayed so as not to be an obstacle. As a result, in the locked state, as in the second embodiment, it is possible to prevent the operator M from obstructing the field of view, thereby improving safety and further improving workability.

<Modification 4>
Also in this embodiment, the locked state may be determined prior to the image display area specifying process. In particular, if the lock display process is a non-display process, or if the locked state is determined, the process ends. As a result, the amount of processing in the locked state can be reduced.

<Modification 5>
Furthermore, this embodiment may be combined with the second embodiment.

<Other Modifications>
Further, in each of the above-described embodiments and modifications, the HMD 300 includes a transmissive display and has been described as an optical see-through type, but is not limited to this. For example, a video see-through type may be used. In this case, the visual field image 380 captured by the line-of-sight camera 319 is displayed on the display 318 as a background. The captured image 370 is synthesized with the image display area 350 of the field-of-view image 380 .

Moreover, the construction machine to which each of the above-described embodiments and modifications is applied is not limited to the deep foundation excavator 100 . In addition to the mechanism for sliding the operation chamber forward, the construction machine may have a mechanism for moving or tilting upward. Moreover, the work device 106 such as the multistage arm 112 and the clamshell bucket 114 may also be various work devices used for construction machinery.

Also, a plurality of imaging devices 200 may be used, and may be a monocular camera, a stereo camera, an infrared camera, or the like.

It should be noted that some or all of the functions and the like of each of the above-described embodiments may be realized by hardware by designing, for example, an integrated circuit. Alternatively, the functions may be realized by software, in which the microprocessor unit or the like interprets and executes a program for realizing each function. Hardware and software may be used together. The software may be stored in a flash memory or the like in advance at the time of product shipment. It may be acquired from various server devices on the Internet after product shipment. Information such as programs, tables, and files for realizing each function may be stored in a recording device such as a memory, a hard disk, or an SSD, or a recording medium such as an IC card, an SD card, or an optical disc. Further, the control lines and information lines indicate those considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. For example, in practice, it may be considered that almost all configurations are interconnected.

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Also, although part of the configuration has been described above, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. is. Moreover, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

100: Construction machine (deep foundation excavator), 101: Construction machine system, 102: Main body, 103: Lower running body, 104: Rotating shaft, 105: Upper rotating body, 106: Work device, 107: Operation room, 108: Machine room, 109: Counterweight, 110: Boom, 111: Boom cylinder, 111A: Pin, 112: Multistage arm, 113: Arm cylinder, 114: Clamshell bucket, 115: Link part, 118: Seat, 119: Operation Lever 119i: Operation lever image 120: Lock lever 121: Operation room front window 124: Seat belt 131: Seating sensor 131i: Floor front edge image 132i: Floor rear edge image
200: photographing device (camera), 211: arithmetic device, 212: RAM, 213: storage device, 214: communication I/F, 215: image sensor, 221: photographed image generation unit, 222: image transmission unit,
300: display device (HMD), 311: arithmetic device, 312: RAM, 313: storage device, 314: communication I/F, 315: input/output I/F, 316: projector, 317: lens, 318: display, 319 : line-of-sight camera 321: image receiving unit 322: relative position calculation unit 323: display image generation unit 324: display unit 325: seating determination unit 326: lock lever state determination unit 331: data storage unit 350 : image display area, 351: four corners, 352: four corners, 353: four corners, 354: four corners, 360: display area, 361: background real image, 370: photographed image, 371: translucent image, 380: field of view image, 390: display picture,
M: operator

Claims (11)

A construction machine comprising a main body for executing work and a photographing device provided outside an operation room of the main body for photographing the surroundings and obtaining a photographed image,
A display device that performs a first display process for displaying the captured image captured by the imaging device in a predetermined image display area of a head mounted display,
The construction machine, wherein the image display area is between left and right operation lever images in a display visual field range that is a visual field range of an operator on the head-mounted display. A construction machine according to claim 1,
The construction machine, wherein the image display area is an area within the display visual field range and further surrounded by a floor front end image of the operation room. The construction machine according to claim 1 or 2,
The construction machine, wherein the image display area is an area within the display visual field range and further surrounded by a floor rear end image of the operation room. The construction machine according to any one of claims 1 to 3,
The display device
further comprising a seating determination unit that determines whether the operator is seated in the operation chamber;
A construction machine characterized in that, when it is determined that the operator is not seated, a second display process different from the first display process is performed. The construction machine according to any one of claims 1 to 4,
The display device
further comprising a lever state determination unit that determines whether the operating lever is in a locked state;
The construction machine, wherein when it is determined that the operation lever is in the locked state, a second display process different from the first display process is performed. The construction machine according to claim 4 or 5,
The construction machine, wherein the second display process is a process of displaying the captured image in a size smaller than that of the first display process. The construction machine according to claim 4 or 5,
The construction machine, wherein the second display processing is processing for displaying the captured image as a translucent image. The construction machine according to claim 4 or 5,
The construction machine, wherein the second display process is a process of moving the captured image to a predetermined position and displaying it. The construction machine according to claim 4 or 5,
The construction machine, wherein the second display process is a process of not displaying the captured image. The construction machine according to any one of claims 1 to 9,
The construction machine, wherein the display of the head-mounted display is an optical see-through type that displays the photographed image superimposed on the actual scenery of the outside world. The construction machine according to any one of claims 1 to 9,
The display of the head mounted display is a video see-through type,
The head mounted display is
a line-of-sight camera that acquires the actual scenery of the outside world in the line-of-sight direction of the operator as a field-of-view video,
A construction machine characterized in that the photographed image is superimposed on the visual field image and displayed.

Images