US20160217331A1 - Monitoring image display device of industrial machine - Google Patents

Monitoring image display device of industrial machine Download PDF

Info

Publication number
US20160217331A1
US20160217331A1 US14/916,058 US201514916058A US2016217331A1 US 20160217331 A1 US20160217331 A1 US 20160217331A1 US 201514916058 A US201514916058 A US 201514916058A US 2016217331 A1 US2016217331 A1 US 2016217331A1
Authority
US
United States
Prior art keywords
image
industrial machine
monitoring
cameras
revolving upperstructure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US14/916,058
Inventor
Yoichi Kowatari
Hidefumi Ishimoto
Moritaka OOTA
Yoshihiro Inanobe
Hiroyoshi Tanaka
Kouji Fujita
Takashi KUSAMA
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Hitachi Construction Machinery Co Ltd
Original Assignee
Hitachi Construction Machinery Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Hitachi Construction Machinery Co Ltd filed Critical Hitachi Construction Machinery Co Ltd
Assigned to HITACHI CONSTRUCTION MACHINERY CO., LTD. reassignment HITACHI CONSTRUCTION MACHINERY CO., LTD. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: KUSAMA, TAKASHI, OOTA, Moritaka, FUJITA, KOUJI, INANOBE, Yoshihiro, ISHIMOTO, HIDEFUMI, KOWATARI, YOICHI, TANAKA, HIROYOSHI
Publication of US20160217331A1 publication Critical patent/US20160217331A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06VIMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00Scenes; Scene-specific elements
    • G06V20/50Context or environment of the image
    • G06V20/56Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06K9/00791
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/20Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/22Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle
    • B60R1/23Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view
    • B60R1/27Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles for viewing an area outside the vehicle, e.g. the exterior of the vehicle with a predetermined field of view providing all-round vision, e.g. using omnidirectional cameras
    • EFIXED CONSTRUCTIONS
    • E02HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02FDREDGING; SOIL-SHIFTING
    • E02F9/00Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/24Safety devices, e.g. for preventing overload
    • EFIXED CONSTRUCTIONS
    • E02HYDRAULIC ENGINEERING; FOUNDATIONS; SOIL SHIFTING
    • E02FDREDGING; SOIL-SHIFTING
    • E02F9/00Component parts of dredgers or soil-shifting machines, not restricted to one of the kinds covered by groups E02F3/00 - E02F7/00
    • E02F9/26Indicating devices
    • E02F9/261Surveying the work-site to be treated
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/002D [Two Dimensional] image generation
    • G06T11/60Editing figures and text; Combining figures or text
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H04N5/247
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00Television systems
    • H04N7/18Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/10Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
    • B60R2300/105Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using multiple cameras
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/30Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
    • B60R2300/304Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
    • B60R2300/305Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images merging camera image with lines or icons
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60RVEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R2300/00Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
    • B60R2300/60Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective
    • B60R2300/607Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by monitoring and displaying vehicle exterior scenes from a transformed perspective from a bird's eye viewpoint

Definitions

  • the present invention relates to a monitoring image display device of an industrial machine, such as, in particular, a large hydraulic excavator for mining, a dump truck, a road construction machine, and the like, to monitor the surroundings of the industrial machine during operation.
  • an industrial machine such as, in particular, a large hydraulic excavator for mining, a dump truck, a road construction machine, and the like.
  • a hydraulic excavator has a lower travel base and a revolving upperstructure which is revolvably coupled to the lower travel base.
  • the revolving upperstructure is equipped with a front working mechanism as a working mechanism.
  • the front working mechanism has a boom and an arm.
  • the boom is mounted to the revolving upperstructure in such a manner as to be raised/lowered.
  • the arm is vertically rotatably coupled to the distal end of the boom.
  • the distal end of the arm is equipped with a front attachment.
  • the front attachment is a bucket when used in excavation work.
  • the surround monitoring device is configured to include a camera mounted to the revolving upperstructure, and a monitor placed, within a cab, in front of the operator seat in which the operator is to sit. Images taken by the camera are displayed on the monitor screen in the form of moving images. Also, ensuring the safety during traveling of the hydraulic excavator and during excavation work or the like, and the like, requires the field of view covering the rear and both the left and right sides of the revolving upperstructure.
  • the revolving upperstructure equipped with a plurality of cameras is conventionally used. This eliminates blind spots from almost all around the revolving upperstructure, ensuring the work safety and improving the maneuverability of the industrial machine.
  • Patent Literature 1 discloses the configuration in which view transformation processing is applied to images of the surroundings of the industrial machine captured by a plurality of cameras for a view from above so that the images are displayed as two-dimensionally-projected top-view monitoring images on the monitor screen.
  • the plurality of cameras are mounted to the revolving upperstructure of the hydraulic excavator and the images of the respective cameras are combined on the monitor screen in order to obtain the surround monitoring image.
  • the cameras are mounted in three locations, in a position in a rear portion of and positions in the right and left side portions of the hydraulic excavator, in which the optical axis of the imaging lens of each of the cameras is directed in an obliquely downward direction.
  • An image taken by each camera is a through image, so that the view transformation processing is applied to the through image to obtain a monitoring image.
  • the view after the transformation is a view from above, thereby obtaining the three top-view images projected from above the hydraulic excavator.
  • the top-view images are displayed on the monitor screen, in which an image illustration symbolically showing the hydraulic excavator (specifically, a plane image illustration of the hydraulic excavator) is displayed on the monitor screen and the top-view images captured by the respective cameras are arranged around the image illustration. More specifically, the image illustration is displayed on a central position of the monitor screen and the top-view images are arranged respectively in regions on the rear side and the left and right sides of the image illustration, resulting in displaying the bird's-eye view image for surround monitoring.
  • an image illustration symbolically showing the hydraulic excavator specifically, a plane image illustration of the hydraulic excavator
  • Such a hydraulic excavator is also used for mining in a mine.
  • the hydraulic excavator for mining which is very large in size, is structured to include a crawler-type lower travel base having left and right crawler belts.
  • the revolving upperstructure installed through a revolving apparatus onto the lower travel base is located at a height of 1.8 meters or more from ground level.
  • the cameras are mounted to the revolving upperstructure and a large empty space exists in a position under the underside of a vehicle frame of the revolving upperstructure.
  • the worker may possibly not be captured within the field of view of the surround-monitoring camera.
  • Enhancing the effectiveness of the monitoring of the surrounding to the industrial machine involves a lessening of the section of blind spot of the monitoring camera, and it is desired for this purpose that the blind spot is absent.
  • Locations of interest to be monitored do not include, understandably, a location at a distance from the industrial machine where the operation of the industrial machine is not affected, a space-limited location where a worker and the like cannot enter, and the tread of the crawler band forming part of the lower travel base which touches the ground.
  • the present invention has been made in light of the aforementioned points and it is an object of the invention to expand the range of the field of view of the monitoring cameras for a further enhancement in the effectiveness of surround monitoring.
  • the monitoring image display device of the industrial machine is a device used in an industrial machine to check for safety of the surroundings of the industrial machine during its operation, and the industrial machine has a movable mechanism mounted to at least a part thereof.
  • the movable mechanism includes a working mechanism, traveling mechanism and/or the like.
  • monitoring using images is performed.
  • the images are displayed on the image display unit, namely, the monitor.
  • As display mode a plane image of the industrial machine is displayed as an icon image.
  • the icon image is represented in an outline.
  • the images of the monitoring cameras are top-view images.
  • the monitoring cameras are located at high elevations so that the optical axis is oriented downward so as to obtain a top-view image.
  • the monitoring cameras are designed to be placed in at least one location or a plurality of locations, and four or more monitoring cameras are preferably mounted in order to photograph the overall surroundings of the industrial machine.
  • the plurality of monitoring cameras are arranged such that their fields of view overlap each other, and the images are trimmed for display with continuity between adjacent images.
  • an empty space is created in an underside position in the outside shape of the industrial machine.
  • a large hydraulic excavator which is an industrial machine, is equipped with a catwalk for the passage of the operator and the like.
  • the catwalk is provided as a structure to jut outward from the frame of the revolving upperstructure. In order to monitor the area under the underside of the catwalk, an image of a section hidden within the icon image is also displayed.
  • the monitoring cameras are sited at high elevations for sections forming the outside shape of the machine, such as the catwalk and the like.
  • the monitoring cameras are placed in positions jutting outward relative to the outline of the structure. This enables displaying a ground surface image including the sections positioned inward of the outside shape.
  • a further increase in the field of view toward the center involves a further increase in height of the camera position to exceed the section defining the outside shape, and also a further increase in length of jutting from the outside shape.
  • an increase in the amount of jutting upward and forward correspondingly increases the possibility of collision with any other substance. Accordingly, the amount of jutting is limited.
  • a revolving radius of the revolving upperstructure is determined as a critical position of the jutting length of the position at which the camera is mounted.
  • a collision with other substance and/or the like must be avoided within the revolving radius of the revolving upperstructure.
  • other substance and the like are prevented from entering the area within the revolving radius.
  • the critical position of the jutting length of the mounting position of the camera is able to be determined within the revolving radius of the revolving upperstructure.
  • a support arm is installed to a predetermined position of a revolving upperstructure main unit forming part of the revolving upperstructure of the hydraulic excavator, so that the camera is mounted to the distal end of the support arm.
  • the support arm may be structured to be adjustable in length in order to make the mounting position of the camera adjustable.
  • strong vibrations may be created such as in the hydraulic excavator, the mounting position of the camera must be prevented from being displaced. Because of this, the support arm is desirably installed in a rigid manner.
  • the hydraulic excavator includes a front working mechanism as a working mechanism installed to the revolving upperstructure for earth/soil excavation.
  • the front working mechanism includes a boom attached to the revolving frame of the revolving upperstructure to be capable of being raised/lowered, an arm coupled to the distal end of the boom to be rotatable in the vertical direction, and a bucket coupled to the distal end of the arm via an articulated mechanism.
  • the bucket is engaged into the ground. Accordingly, during excavation operation, the section in which the bucket makes contact with the ground is a danger region which any other vehicle and worker must be prevented from entering.
  • FIG. 1 is a front view of a hydraulic excavator as an example of the industrial machine.
  • FIG. 2 is a top view of the hydraulic excavator in FIG. 1 .
  • FIG. 3 is a left side view of the hydraulic excavator in FIG. 1 .
  • FIG. 4 is a schematic illustration of the field of view of cameras when viewed from a position in FIG. 3 .
  • FIG. 5 is a schematic illustration of the field of view of cameras when viewed from a position in FIG. 2 .
  • FIG. 6 is a diagram illustrating an example of a display region of each image on the screen of a monitor.
  • FIG. 7 is a top view illustrating an example of an icon image displayed on the monitor.
  • FIG. 8 is a block diagram illustrating the configuration of a display controller of the monitor.
  • FIG. 9 is an illustration describing the principles for making a hidden area under the underside of a catwalk visible to the camera to fall within the angle of view of the camera.
  • FIG. 10 is an illustration showing another example mode of an image on a monitor including an excavatable range by a bucket.
  • FIG. 1 to FIG. 3 illustrate the structure of a hydraulic excavator 1 as an example of the industrial machine.
  • FIG. 1 is a front view
  • FIG. 2 is a top view
  • FIG. 3 is a left side view of the hydraulic excavator 1 .
  • a front working mechanism 5 which will be described later is omitted in FIG. 2 and FIG. 3 .
  • the hydraulic excavator 1 has a lower travel base 2 having a crawler traveling mechanism, and a revolving upperstructure 3 mounted revolvably on the lower travel base 2 .
  • the revolving upperstructure 3 is equipped with a cab 4 for an operator aboard the hydraulic excavator to operate the machine, and the front working mechanism 5 is mounted as a working mechanism for work of earth/soil excavation and/or the like.
  • An illustration and details of the front working mechanism 5 are omitted because the specific structure of the front working mechanism is well known.
  • the hydraulic excavator 1 which is the industrial machine operates the lower travel base 2 for traveling, the revolving upperstructure 3 for revolving and the front working mechanism 5 to perform the work.
  • Each of the above-described operating parts is driven by a hydraulic cylinder and/or a hydraulic motor, in which a movable section and a driving section to move the movable section form a movable mechanism.
  • any other actuators such as an electric motor and/or the like may be used.
  • the movable mechanism of the hydraulic excavator 1 is operated by the operator, and the cab 4 is installed in a forward position of the revolving upperstructure 3 .
  • a facility 6 including a machine room is located rearward of the cab 4 and the front working mechanism 5 which are arranged in an approximately side-by-side position on the revolving upperstructure 3 to which the front working mechanism 5 is mounted, thus forming a revolving upperstructure main unit 3 a .
  • a counterweight 7 is placed at the tail end.
  • the front working mechanism 5 includes a boom attached to the revolving upperstructure main unit 3 a of the revolving upperstructure 3 to be capable of being raised/lowered, an arm coupled to the distal end of the boom to rotate in the vertical direction, and a bucket rotatably coupled to the distal end of the arm for work on earth/soil excavation and/or the like, the boom, the arm and the bucket being driven by the hydraulic cylinders, which are omitted from the drawings. Note that the specific structure of the front working mechanism 5 is well-known in the art.
  • the revolving upperstructure main unit 3 a is further equipped with a ladder 13 for the operator to get into/off the cab 4 , and also a catwalk 14 forming a passage.
  • the catwalk 14 is placed on the side of the revolving upperstructure main unit 3 a and in a midpoint position in the vertical direction to be protruded in the lateral direction so that a position under the underside of the catwalk 14 is empty space.
  • the ladder 13 is placed to be connected to the catwalk 14 .
  • the revolving upperstructure main unit 3 a is formed of a frame structure, and the catwalk 14 and the ladder 13 are the jutting components extending out from the revolving upperstructure main unit 3 a.
  • the hydraulic excavator 1 having the movable mechanisms as described earlier, is equipped with a surround monitoring device for ensuring safety in the surrounding under operation conditions of the movable mechanism.
  • the surround monitoring device is mounted to the revolving upperstructure 3 and includes a plurality of video cameras capable of shooting moving images. More specifically, in FIG. 4 and FIG. 5 , reference sign 15 F denotes a front camera, reference sigh 15 B denotes a rear camera, and reference signs 15 L and 15 R denote left-side and right-side cameras.
  • the surrounding of the hydraulic excavator 1 is monitored through the cameras 15 F, 15 B, 15 L and 15 R.
  • reference sign 15 is used when collectively calling the camera for camera-monitoring.
  • the cameras 15 F, 15 B, 15 L and 15 R are mounted to the revolving upperstructure main unit 3 a of the revolving upperstructure 3 , and are placed at high elevations so that the optical axis of each camera is oriented in the downward or obliquely downward direction.
  • FIG. 4 an image captured by each of the cameras 15 results in a top-view image looking downwardly from above.
  • Installation positions of the respective cameras 15 are determined to have a positional relationship to monitor all around the hydraulic excavator 1 .
  • FIG. 5 illustrates example placement of the cameras 15 F, 15 B, 15 L and 15 R.
  • the image captured by each camera 15 F, 15 B, 15 L, 15 R is displayed on a monitor 20 serving as an image displaying unit as illustrated in FIG. 6 .
  • the monitor 20 is placed in a position within the cab 4 to allow the seated operator to visually check the monitor 20 from the operator's seat where the operator operates the hydraulic excavator 1 .
  • the images sent from the respective cameras 15 F, 15 B, 15 L and 15 R to be displayed on the monitor 20 are top-view images.
  • the screen of the monitor 20 is split into a plurality of regions, such that the above-described top-view images are respectively trimmed to fit into the predetermined regions.
  • camera images 21 F, 21 B, 21 L and 21 R are displayed on the respective display regions set in the monitor 20 .
  • a display region for an icon image 21 C is set at the center of the screen, and a planar image of the hydraulic excavator 1 is symbolized and displayed in the display region of the icon image 21 C as illustrated in FIG. 7 .
  • the icon image 21 C results from simplifying the hydraulic excavator which is the industrial machine of interest for monitoring, and in the drawing, the revolving upperstructure 3 , the front working mechanism 5 and the catwalk 14 of the hydraulic excavator 1 are displayed. These components are designated with reference signs in FIG. 9 .
  • FIG. 8 shows the configuration of a display controller 30 to control displaying of images in the monitor 20 .
  • the display controller 30 has an image correction unit 31 .
  • the image correction unit 31 performs common image signal processing, including aberration correction, contrast correction, color correction and/or the like, on the camera images sent from the respective cameras 15 F, 15 B, 15 L and 15 R, on the basis of camera optical system parameters and/or the like in order to improve the quality of each incoming image.
  • the display controller 30 includes an image trimming unit 32 , in which each of the camera images 21 F, 21 B, 21 L and 21 R captured by the respective cameras 15 F, 15 B, 15 L and 15 R is trimmed to a size and a shape to be displayed in the corresponding display region.
  • the icon image 21 is displayed together with the camera images 21 F, 21 B, 21 L and 21 R of the four cameras.
  • the icon image 21 C is generated by an icon image generating unit 33 .
  • the individual camera images 21 F, 21 B, 21 L and 21 R and the icon image 21 C are composited for displaying by an image compositing unit 33 .
  • the icon image 21 C is displayed on a central position of the monitor 20 , and the individual camera images 21 F, 21 B, 21 L and 21 R are arranged around the icon image 21 C.
  • the output data from the image compositing unit 34 is sent to a display image generating unit 35 . By the output from the display image generating unit 34 , the above-described images are assigned respectively to the corresponding positions to be displayed.
  • the icon image 21 C is also an image representing the hydraulic excavator 1 in a planar form. Therefore, the hydraulic excavator 1 and its surrounding circumstances are displayed in a planar form on the monitor 20 , enabling the operator to recognize the circumstances during operation of the hydraulic excavator 1 with ease.
  • Obstructions such as various members or mechanisms may be located in optical paths from the cameras 15 mounted to the revolving upperstructure 3 to the ground. Accordingly, images corresponding to the positions under such obstructions cannot be obtained. For example, regarding a section on the left side of the revolving upperstructure 3 on which the cab 4 is placed, there is a large empty space between the ground contacting the crawler track belt which constitutes the tread of the lower travel base 2 and the position of the underside of the cab 4 . Further, because the catwalk 14 and the ladder 13 are located rearward of the cab 4 , there is a wide empty space under the undersides of them.
  • the empty spaces contain a spacious hidden area existing due to the cab 4 , catwalk 14 , ladder 13 and/or the like.
  • the operator, other workers and/or the like may possibly enter such hidden-area spaces, but also tools and other equipment may be placed within such hidden-area spaces.
  • a support arm 40 L is provided to jut from the revolving upperstructure main unit 3 a in the lateral direction, so that the camera 15 L photographing the region is not mounted to the revolving upperstructure main unit 3 a and instead the camera 15 L is supported at the distal end of the support arm 40 L.
  • the optical axis of the camera 15 L mounted at the support arm 40 L is located outward of the outer side of the catwalk 14 by dimension p.
  • the camera 15 L is located at a height h above the ground. Because of this, at a level of and around the ground surface, the camera 15 L is able to have a field of view covering virtually the crawler belt forming part of the lower travel base 2 .
  • the field of view of the camera 15 L is able to cover the position under the underside of the catwalk 14 .
  • the right camera 15 R and the front camera 15 F are mounted to support arms 40 R and 40 F each of which juts from the revolving upperstructure main unit 3 a in a predetermined direction, increasing the field of view for the sections.
  • the character image 21 C is displayed with a range resulting from an increase in the field of view being in a transparent state.
  • the symbol image of the hydraulic excavator 1 displayed in the icon image 21 C includes a part in the transparent state, but the outside shape of the hydraulic excavator 1 is displayed in outline. Accordingly, in the following description, reference sign 40 is used to correctively call the support arm.
  • the installation position and the jutting length of the support arm 40 can be determined as appropriate on the basis of the structure of the revolving upperstructure 3 .
  • the jutting length is determined with reference to a revolving radius R shown in FIG. 2 , and as long as the jutting length is within the revolving radius R, even if the support arm 40 juts greatly from the revolving upperstructure main unit 3 a , there is no possibility of the support arm 40 coming into collision with other substance, and the like.
  • the support arm 40 is preferably formed of rigid body, so that the camera 15 is firmly fixed to the support arm 40 in order to prevent the camera 15 from being displaced from its normal position by vibration and/or the like.
  • the support arm 40 L for the camera 15 L mounted on the left side extends in the lateral direction and the support arms 40 F and 40 R for the cameras 15 F and 15 R extend in the forward direction, but the extending direction can be determined based on the position of the hidden area to be displayed on the monitor 20 .
  • the camera 15 B is located in a position on an upper portion of the counterweight 7 without an extra support arm.
  • the camera 15 B may be supported by a support arm.
  • the structure designed as described above enables displaying of the hidden sections covered with the revolving upperstructure 3 , such as the position under the underside of the catwalk 14 and the like, as well as the sections of the surrounding of the hydraulic excavator 1 on the monitor 20 .
  • the revolving upperstructure 3 with the camera 15 mounted thereto even if a section is out of the field of view, the section where a worker and the like may possibly enter is able to be displayed on the monitor 20 , enhancing the effectiveness as the surround monitoring.
  • the hidden area transparently displayed is a section where a person and the like may possibly enter. There's no need to display, for example, a section where a person and the like may not enter such as the tread of the crawler belt and the like. Accordingly, the level position and the jutting length of the support arm 40 to which the camera 15 is mounted may be determined as appropriate on the basis of the position and expanse of a hidden area which is required to be transparently displayed.
  • the hydraulic excavator 1 is used for work on earth/soil excavation and the like, and the front working mechanism 5 is operated.
  • the front working mechanism 5 When the front working mechanism 5 is operated for earth/soil excavation, the bucket attached at the distal end is driven from above to cut deep into the earth. Because of this, if a person or a substance is located within the range of motion track of the bucket, the bucket may possibly collide with the person or substance.
  • the monitor 20 is designed to display, as an excavatable range B, a range in which the bucket can make contact with the ground within the reach range of the front working mechanism 5 .
  • the front working mechanism 5 is mounted to the revolving upperstructure 3 , so that the front working mechanism 5 revolves in synchronism with the revolving upperstructure 3 when the revolving upperstructure 3 is operated to revolve. Then, the monitoring cameras 15 F, 15 B, 15 L and 15 R are also mounted to the revolving upperstructure 3 , so that no change occurs in the positional relationship with the front working mechanism 5 on the images captured by the cameras 15 F, 15 B, 15 L and 15 R.
  • the buckets are of a plurality of types of varying width dimension. For this reason, the excavatable range B is desirably varied according to the structure of the front working mechanism 5 actually mounted to the hydraulic excavator 12 .
  • displaying an excavatable range B on the monitor 20 enables the driving of the hydraulic excavator 1 with reference to the excavatable range B when the hydraulic excavator 1 is travelled and revolved to select an excavation site, facilitating positional adjustment for excavation site.
  • displaying the excavatable range B on the monitor 20 offers enhanced work efficiencies as well as the ensuring of work safety.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Mining & Mineral Resources (AREA)
  • Civil Engineering (AREA)
  • General Engineering & Computer Science (AREA)
  • Structural Engineering (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Component Parts Of Construction Machinery (AREA)
  • Closed-Circuit Television Systems (AREA)

Abstract

In an industrial machine, the work safety is further improved by eliminating blind spots from an image displayed on a monitor. A hydraulic excavator 1 which is the industrial machine is equipped with monitoring cameras 15F, 15B, 15L, 15R mounted in the respective places of a revolving upperstructure 3 in order to capture images for monitoring. A monitor 20 displays camera images 21F, 21B, 21L, 21R obtained by the cameras as well as an icon image 21C of an image illustration of the hydraulic excavator 1. The cameras 15F, 15L, 15R are mounted at the distal ends of support arms 40F, 40L, 40R to be located in positions jutting from a revolving upperstructure main unit 3 a of the revolving upperstructure 3, so that a hidden area under the underside of a catwalk 14 provided on the revolving upperstructure 3 falls within the field of view.

Description

    TECHNICAL FIELD
  • The present invention relates to a monitoring image display device of an industrial machine, such as, in particular, a large hydraulic excavator for mining, a dump truck, a road construction machine, and the like, to monitor the surroundings of the industrial machine during operation.
  • BACKGROUND ART
  • As the industrial machine, for example, a hydraulic excavator has a lower travel base and a revolving upperstructure which is revolvably coupled to the lower travel base. The revolving upperstructure is equipped with a front working mechanism as a working mechanism. The front working mechanism has a boom and an arm. The boom is mounted to the revolving upperstructure in such a manner as to be raised/lowered. The arm is vertically rotatably coupled to the distal end of the boom. The distal end of the arm is equipped with a front attachment. The front attachment is a bucket when used in excavation work.
  • It is well known in the art to provide a surround monitoring device to monitor the surrounding circumstances of the revolving upperstructure in order to ensure work safety while the front working mechanism serving as the working mechanism is driven to carry out earth/soil excavation work and/or the like. The surround monitoring device is configured to include a camera mounted to the revolving upperstructure, and a monitor placed, within a cab, in front of the operator seat in which the operator is to sit. Images taken by the camera are displayed on the monitor screen in the form of moving images. Also, ensuring the safety during traveling of the hydraulic excavator and during excavation work or the like, and the like, requires the field of view covering the rear and both the left and right sides of the revolving upperstructure. In order to provide a wide-angle field of view around the hydraulic excavator which is the industrial machinery, the revolving upperstructure equipped with a plurality of cameras is conventionally used. This eliminates blind spots from almost all around the revolving upperstructure, ensuring the work safety and improving the maneuverability of the industrial machine.
  • Patent Literature 1 discloses the configuration in which view transformation processing is applied to images of the surroundings of the industrial machine captured by a plurality of cameras for a view from above so that the images are displayed as two-dimensionally-projected top-view monitoring images on the monitor screen. With the configuration in Patent Literature 1, the plurality of cameras are mounted to the revolving upperstructure of the hydraulic excavator and the images of the respective cameras are combined on the monitor screen in order to obtain the surround monitoring image.
  • The cameras are mounted in three locations, in a position in a rear portion of and positions in the right and left side portions of the hydraulic excavator, in which the optical axis of the imaging lens of each of the cameras is directed in an obliquely downward direction. An image taken by each camera is a through image, so that the view transformation processing is applied to the through image to obtain a monitoring image. The view after the transformation is a view from above, thereby obtaining the three top-view images projected from above the hydraulic excavator.
  • With this configuration, the top-view images are displayed on the monitor screen, in which an image illustration symbolically showing the hydraulic excavator (specifically, a plane image illustration of the hydraulic excavator) is displayed on the monitor screen and the top-view images captured by the respective cameras are arranged around the image illustration. More specifically, the image illustration is displayed on a central position of the monitor screen and the top-view images are arranged respectively in regions on the rear side and the left and right sides of the image illustration, resulting in displaying the bird's-eye view image for surround monitoring.
  • CITATION LIST Patent Literature
    • Patent Literature 1: Japanese Patent Application Laid-Open No. 2010-204821
    SUMMARY OF INVENTION Technical Problem
  • Such a hydraulic excavator is also used for mining in a mine. The hydraulic excavator for mining, which is very large in size, is structured to include a crawler-type lower travel base having left and right crawler belts. The revolving upperstructure installed through a revolving apparatus onto the lower travel base is located at a height of 1.8 meters or more from ground level. In this manner, the cameras are mounted to the revolving upperstructure and a large empty space exists in a position under the underside of a vehicle frame of the revolving upperstructure. In the situation in which a worker working around the hydraulic excavator enters the space under the underside of the vehicle frame of the revolving upperstructure, the worker may possibly not be captured within the field of view of the surround-monitoring camera.
  • Enhancing the effectiveness of the monitoring of the surrounding to the industrial machine involves a lessening of the section of blind spot of the monitoring camera, and it is desired for this purpose that the blind spot is absent. Locations of interest to be monitored do not include, understandably, a location at a distance from the industrial machine where the operation of the industrial machine is not affected, a space-limited location where a worker and the like cannot enter, and the tread of the crawler band forming part of the lower travel base which touches the ground. However, especially, for a large/heavy industrial machine, there is a necessity to cause the field of view of the monitor cameras to cover the areas under the underside of the frame of the revolving upperstructure and the section extending out from the frame to the extent possible.
  • The present invention has been made in light of the aforementioned points and it is an object of the invention to expand the range of the field of view of the monitoring cameras for a further enhancement in the effectiveness of surround monitoring.
  • Solution to Problem
  • To address the aforementioned technical problems, the present invention provides a monitoring image display device installed on an industrial machine which has a movable mechanism provided at least in a part, the monitoring image display device including: one or more monitoring cameras which are placed in a location at least partially extending beyond an outline region to have a field of view covering the surroundings of the industrial machine; an image generating unit that generates top-view images captured by the monitoring cameras looking downward from above; an icon-image generating unit that generates an icon image representing a plane of the industrial machine with at least the outline being shown in a graphic form; an image compositing unit that performs composite processing to display an image captured by the image generating unit around the icon image generated by the icon-image generating unit; and an image display unit that displays the top-view images composited by the image compositing unit to include part of a section hidden within a display region of the icon image.
  • The monitoring image display device of the industrial machine according to the present invention is a device used in an industrial machine to check for safety of the surroundings of the industrial machine during its operation, and the industrial machine has a movable mechanism mounted to at least a part thereof. Specifically, the movable mechanism includes a working mechanism, traveling mechanism and/or the like. For safe operation of the movable mechanism structured as described above, monitoring using images is performed. The images are displayed on the image display unit, namely, the monitor. As display mode, a plane image of the industrial machine is displayed as an icon image. The icon image is represented in an outline.
  • Images captured by the monitoring cameras, together with the icon image, are displayed. Therefore, the images of the monitoring cameras are top-view images. For the purpose, the monitoring cameras are located at high elevations so that the optical axis is oriented downward so as to obtain a top-view image. The monitoring cameras are designed to be placed in at least one location or a plurality of locations, and four or more monitoring cameras are preferably mounted in order to photograph the overall surroundings of the industrial machine. The plurality of monitoring cameras are arranged such that their fields of view overlap each other, and the images are trimmed for display with continuity between adjacent images.
  • Because of the structure of the industrial machine, an empty space is created in an underside position in the outside shape of the industrial machine. For example, a large hydraulic excavator, which is an industrial machine, is equipped with a catwalk for the passage of the operator and the like. The catwalk is provided as a structure to jut outward from the frame of the revolving upperstructure. In order to monitor the area under the underside of the catwalk, an image of a section hidden within the icon image is also displayed.
  • The monitoring cameras are sited at high elevations for sections forming the outside shape of the machine, such as the catwalk and the like. In addition, the monitoring cameras are placed in positions jutting outward relative to the outline of the structure. This enables displaying a ground surface image including the sections positioned inward of the outside shape. A further increase in the field of view toward the center involves a further increase in height of the camera position to exceed the section defining the outside shape, and also a further increase in length of jutting from the outside shape. However, an increase in the amount of jutting upward and forward correspondingly increases the possibility of collision with any other substance. Accordingly, the amount of jutting is limited.
  • In a machine with a revolving upperstructure revolvably mounted on a lower travel base as in the case of the hydraulic excavator, a revolving radius of the revolving upperstructure is determined as a critical position of the jutting length of the position at which the camera is mounted. For safe revolving, a collision with other substance and/or the like must be avoided within the revolving radius of the revolving upperstructure. During operation, other substance and the like are prevented from entering the area within the revolving radius. As a result, the critical position of the jutting length of the mounting position of the camera is able to be determined within the revolving radius of the revolving upperstructure. Then, a support arm is installed to a predetermined position of a revolving upperstructure main unit forming part of the revolving upperstructure of the hydraulic excavator, so that the camera is mounted to the distal end of the support arm. The support arm may be structured to be adjustable in length in order to make the mounting position of the camera adjustable. However, if strong vibrations may be created such as in the hydraulic excavator, the mounting position of the camera must be prevented from being displaced. Because of this, the support arm is desirably installed in a rigid manner.
  • Next, the hydraulic excavator includes a front working mechanism as a working mechanism installed to the revolving upperstructure for earth/soil excavation. The front working mechanism includes a boom attached to the revolving frame of the revolving upperstructure to be capable of being raised/lowered, an arm coupled to the distal end of the boom to be rotatable in the vertical direction, and a bucket coupled to the distal end of the arm via an articulated mechanism. During excavation, the bucket is engaged into the ground. Accordingly, during excavation operation, the section in which the bucket makes contact with the ground is a danger region which any other vehicle and worker must be prevented from entering.
  • For this reason, when the hydraulic excavator is in a predetermined position, that is, when the revolving upperstructure is in a position at a predetermined angle, it is important in terms of work safety that the region in which excavation using the bucket will be carried out is displayed on the monitor. In particular, when doing the excavation work by remote control without the operator aboard, displaying the location of interest for excavation on the monitor is critical in terms of work safety and efficiency.
  • Advantageous Effects of Invention
  • In the industrial machine such as the hydraulic excavator and the like, a decrease in blind spots in an image displayed on the monitor provides a higher level of work safety.
  • BRIEF DESCRIPTION OF DRAWINGS
  • FIG. 1 is a front view of a hydraulic excavator as an example of the industrial machine.
  • FIG. 2 is a top view of the hydraulic excavator in FIG. 1.
  • FIG. 3 is a left side view of the hydraulic excavator in FIG. 1.
  • FIG. 4 is a schematic illustration of the field of view of cameras when viewed from a position in FIG. 3.
  • FIG. 5 is a schematic illustration of the field of view of cameras when viewed from a position in FIG. 2.
  • FIG. 6 is a diagram illustrating an example of a display region of each image on the screen of a monitor.
  • FIG. 7 is a top view illustrating an example of an icon image displayed on the monitor.
  • FIG. 8 is a block diagram illustrating the configuration of a display controller of the monitor.
  • FIG. 9 is an illustration describing the principles for making a hidden area under the underside of a catwalk visible to the camera to fall within the angle of view of the camera.
  • FIG. 10 is an illustration showing another example mode of an image on a monitor including an excavatable range by a bucket.
  • DESCRIPTION OF EMBODIMENTS
  • Embodiments according to the present invention will now be described with reference to the accompanying drawings. First, FIG. 1 to FIG. 3 illustrate the structure of a hydraulic excavator 1 as an example of the industrial machine. FIG. 1 is a front view, FIG. 2 is a top view and FIG. 3 is a left side view of the hydraulic excavator 1. Here, a front working mechanism 5 which will be described later is omitted in FIG. 2 and FIG. 3.
  • The hydraulic excavator 1 has a lower travel base 2 having a crawler traveling mechanism, and a revolving upperstructure 3 mounted revolvably on the lower travel base 2. The revolving upperstructure 3 is equipped with a cab 4 for an operator aboard the hydraulic excavator to operate the machine, and the front working mechanism 5 is mounted as a working mechanism for work of earth/soil excavation and/or the like. An illustration and details of the front working mechanism 5 are omitted because the specific structure of the front working mechanism is well known.
  • Thus, the hydraulic excavator 1 which is the industrial machine operates the lower travel base 2 for traveling, the revolving upperstructure 3 for revolving and the front working mechanism 5 to perform the work. Each of the above-described operating parts is driven by a hydraulic cylinder and/or a hydraulic motor, in which a movable section and a driving section to move the movable section form a movable mechanism. Also, any other actuators such as an electric motor and/or the like may be used.
  • The movable mechanism of the hydraulic excavator 1 is operated by the operator, and the cab 4 is installed in a forward position of the revolving upperstructure 3. On the right side of the cab 4, a facility 6 including a machine room is located rearward of the cab 4 and the front working mechanism 5 which are arranged in an approximately side-by-side position on the revolving upperstructure 3 to which the front working mechanism 5 is mounted, thus forming a revolving upperstructure main unit 3 a. Further, a counterweight 7 is placed at the tail end.
  • The front working mechanism 5 includes a boom attached to the revolving upperstructure main unit 3 a of the revolving upperstructure 3 to be capable of being raised/lowered, an arm coupled to the distal end of the boom to rotate in the vertical direction, and a bucket rotatably coupled to the distal end of the arm for work on earth/soil excavation and/or the like, the boom, the arm and the bucket being driven by the hydraulic cylinders, which are omitted from the drawings. Note that the specific structure of the front working mechanism 5 is well-known in the art.
  • The revolving upperstructure main unit 3 a is further equipped with a ladder 13 for the operator to get into/off the cab 4, and also a catwalk 14 forming a passage. The catwalk 14 is placed on the side of the revolving upperstructure main unit 3 a and in a midpoint position in the vertical direction to be protruded in the lateral direction so that a position under the underside of the catwalk 14 is empty space. Further, the ladder 13 is placed to be connected to the catwalk 14. In this connection, the revolving upperstructure main unit 3 a is formed of a frame structure, and the catwalk 14 and the ladder 13 are the jutting components extending out from the revolving upperstructure main unit 3 a.
  • The hydraulic excavator 1, having the movable mechanisms as described earlier, is equipped with a surround monitoring device for ensuring safety in the surrounding under operation conditions of the movable mechanism. The surround monitoring device is mounted to the revolving upperstructure 3 and includes a plurality of video cameras capable of shooting moving images. More specifically, in FIG. 4 and FIG. 5, reference sign 15F denotes a front camera, reference sigh 15B denotes a rear camera, and reference signs 15L and 15R denote left-side and right-side cameras. The surrounding of the hydraulic excavator 1 is monitored through the cameras 15F, 15B, 15L and 15R. Incidentally, in the following description, reference sign 15 is used when collectively calling the camera for camera-monitoring.
  • The cameras 15F, 15B, 15L and 15R are mounted to the revolving upperstructure main unit 3 a of the revolving upperstructure 3, and are placed at high elevations so that the optical axis of each camera is oriented in the downward or obliquely downward direction. Thus, as shown in FIG. 4, an image captured by each of the cameras 15 results in a top-view image looking downwardly from above. Installation positions of the respective cameras 15 are determined to have a positional relationship to monitor all around the hydraulic excavator 1. FIG. 5 illustrates example placement of the cameras 15F, 15B, 15L and 15R.
  • As described above, the image captured by each camera 15F, 15B, 15L, 15R is displayed on a monitor 20 serving as an image displaying unit as illustrated in FIG. 6. Here, the monitor 20 is placed in a position within the cab 4 to allow the seated operator to visually check the monitor 20 from the operator's seat where the operator operates the hydraulic excavator 1.
  • The images sent from the respective cameras 15F, 15B, 15L and 15R to be displayed on the monitor 20 are top-view images. The screen of the monitor 20 is split into a plurality of regions, such that the above-described top-view images are respectively trimmed to fit into the predetermined regions. As a result, camera images 21F, 21B, 21L and 21R are displayed on the respective display regions set in the monitor 20. Then, a display region for an icon image 21C is set at the center of the screen, and a planar image of the hydraulic excavator 1 is symbolized and displayed in the display region of the icon image 21C as illustrated in FIG. 7. Here, the icon image 21C results from simplifying the hydraulic excavator which is the industrial machine of interest for monitoring, and in the drawing, the revolving upperstructure 3, the front working mechanism 5 and the catwalk 14 of the hydraulic excavator 1 are displayed. These components are designated with reference signs in FIG. 9.
  • FIG. 8 shows the configuration of a display controller 30 to control displaying of images in the monitor 20. The display controller 30 has an image correction unit 31. The image correction unit 31 performs common image signal processing, including aberration correction, contrast correction, color correction and/or the like, on the camera images sent from the respective cameras 15F, 15B, 15L and 15R, on the basis of camera optical system parameters and/or the like in order to improve the quality of each incoming image.
  • Because the monitor 20 provides the assignment display using the display regions for the camera images 21F, 21B, 21L and 21R, the display controller 30 includes an image trimming unit 32, in which each of the camera images 21F, 21B, 21L and 21R captured by the respective cameras 15F, 15B, 15L and 15R is trimmed to a size and a shape to be displayed in the corresponding display region.
  • On the monitor 20 the icon image 21 is displayed together with the camera images 21F, 21B, 21L and 21R of the four cameras. The icon image 21C is generated by an icon image generating unit 33. Then, the individual camera images 21F, 21B, 21L and 21R and the icon image 21C are composited for displaying by an image compositing unit 33. The icon image 21C is displayed on a central position of the monitor 20, and the individual camera images 21F, 21B, 21L and 21R are arranged around the icon image 21C. The output data from the image compositing unit 34 is sent to a display image generating unit 35. By the output from the display image generating unit 34, the above-described images are assigned respectively to the corresponding positions to be displayed.
  • Because the cameras 15F, 15B, 15L and 15R photograph with their optical axes directed in an approximately vertical direction, all the camera images thus captured are top-view images. The icon image 21C is also an image representing the hydraulic excavator 1 in a planar form. Therefore, the hydraulic excavator 1 and its surrounding circumstances are displayed in a planar form on the monitor 20, enabling the operator to recognize the circumstances during operation of the hydraulic excavator 1 with ease.
  • Obstructions such as various members or mechanisms may be located in optical paths from the cameras 15 mounted to the revolving upperstructure 3 to the ground. Accordingly, images corresponding to the positions under such obstructions cannot be obtained. For example, regarding a section on the left side of the revolving upperstructure 3 on which the cab 4 is placed, there is a large empty space between the ground contacting the crawler track belt which constitutes the tread of the lower travel base 2 and the position of the underside of the cab 4. Further, because the catwalk 14 and the ladder 13 are located rearward of the cab 4, there is a wide empty space under the undersides of them. This means that, when viewing the revolving upperstructure 3 in top view, the empty spaces contain a spacious hidden area existing due to the cab 4, catwalk 14, ladder 13 and/or the like. In addition, not only the operator, other workers and/or the like may possibly enter such hidden-area spaces, but also tools and other equipment may be placed within such hidden-area spaces. Further, in a heavier hydraulic excavator, there is room enough for accommodating a vehicle and/or the like. Accordingly, in terms of monitoring around the hydraulic excavator 1, it is desirable that the above-described empty space of the hidden area falls within the field of view of the monitoring camera.
  • Given these circumstances, as illustrated in FIG. 9, a support arm 40L is provided to jut from the revolving upperstructure main unit 3 a in the lateral direction, so that the camera 15L photographing the region is not mounted to the revolving upperstructure main unit 3 a and instead the camera 15L is supported at the distal end of the support arm 40L. Accordingly, the optical axis of the camera 15L mounted at the support arm 40L is located outward of the outer side of the catwalk 14 by dimension p. Further, the camera 15L is located at a height h above the ground. Because of this, at a level of and around the ground surface, the camera 15L is able to have a field of view covering virtually the crawler belt forming part of the lower travel base 2. In addition, because the camera 15L is placed forward of the position of the catwalk 14 with the ladder 13 mounted thereto, the field of view of the camera 15L is able to cover the position under the underside of the catwalk 14.
  • Likewise, the right camera 15R and the front camera 15F are mounted to support arms 40R and 40F each of which juts from the revolving upperstructure main unit 3 a in a predetermined direction, increasing the field of view for the sections. Further, on the monitor 20, the character image 21C is displayed with a range resulting from an increase in the field of view being in a transparent state. In this case, the symbol image of the hydraulic excavator 1 displayed in the icon image 21C includes a part in the transparent state, but the outside shape of the hydraulic excavator 1 is displayed in outline. Accordingly, in the following description, reference sign 40 is used to correctively call the support arm.
  • In this manner, mounting the camera 15 to the distal end of the support arm 40 provided to jut from the revolving upperstructure main unit 3 a enables the field of view covering the hidden area of the revolving upperstructure 3. In this respect, the installation position and the jutting length of the support arm 40 can be determined as appropriate on the basis of the structure of the revolving upperstructure 3. The jutting length is determined with reference to a revolving radius R shown in FIG. 2, and as long as the jutting length is within the revolving radius R, even if the support arm 40 juts greatly from the revolving upperstructure main unit 3 a, there is no possibility of the support arm 40 coming into collision with other substance, and the like. The support arm 40 is preferably formed of rigid body, so that the camera 15 is firmly fixed to the support arm 40 in order to prevent the camera 15 from being displaced from its normal position by vibration and/or the like.
  • Here, in an example embodiment according to the present invention, the support arm 40L for the camera 15L mounted on the left side extends in the lateral direction and the support arms 40F and 40R for the cameras 15F and 15R extend in the forward direction, but the extending direction can be determined based on the position of the hidden area to be displayed on the monitor 20. In the illustrated case, the camera 15B is located in a position on an upper portion of the counterweight 7 without an extra support arm. However, the camera 15B may be supported by a support arm.
  • The structure designed as described above enables displaying of the hidden sections covered with the revolving upperstructure 3, such as the position under the underside of the catwalk 14 and the like, as well as the sections of the surrounding of the hydraulic excavator 1 on the monitor 20. In addition, in the revolving upperstructure 3 with the camera 15 mounted thereto, even if a section is out of the field of view, the section where a worker and the like may possibly enter is able to be displayed on the monitor 20, enhancing the effectiveness as the surround monitoring.
  • In the position under the underside of the revolving upperstructure main unit 3 a forming part of the revolving upperstructure 3, the hidden area transparently displayed is a section where a person and the like may possibly enter. There's no need to display, for example, a section where a person and the like may not enter such as the tread of the crawler belt and the like. Accordingly, the level position and the jutting length of the support arm 40 to which the camera 15 is mounted may be determined as appropriate on the basis of the position and expanse of a hidden area which is required to be transparently displayed.
  • On the other hand, the hydraulic excavator 1 is used for work on earth/soil excavation and the like, and the front working mechanism 5 is operated. When the front working mechanism 5 is operated for earth/soil excavation, the bucket attached at the distal end is driven from above to cut deep into the earth. Because of this, if a person or a substance is located within the range of motion track of the bucket, the bucket may possibly collide with the person or substance. During the work, there is a necessity to protect the person, substance and/or the like from collision with the front working mechanism 5. Naturally, the operator aboard the cab 4 can visually recognize the motion of the front working mechanism. Given these circumstances, as illustrated in FIG. 10, the monitor 20 is designed to display, as an excavatable range B, a range in which the bucket can make contact with the ground within the reach range of the front working mechanism 5.
  • The front working mechanism 5 is mounted to the revolving upperstructure 3, so that the front working mechanism 5 revolves in synchronism with the revolving upperstructure 3 when the revolving upperstructure 3 is operated to revolve. Then, the monitoring cameras 15F, 15B, 15L and 15R are also mounted to the revolving upperstructure 3, so that no change occurs in the positional relationship with the front working mechanism 5 on the images captured by the cameras 15F, 15B, 15L and 15R. Therefore, an excavatable range B in which the front working mechanism 5 can make contact with the ground, together with the camera images 21F, 21B, 21L and 21R arranged around the icon image 21C, is displayed on the monitor 20, in which the excavatable range B is determined on the basis of the width dimension of the bucket and the maximum reach length. The buckets are of a plurality of types of varying width dimension. For this reason, the excavatable range B is desirably varied according to the structure of the front working mechanism 5 actually mounted to the hydraulic excavator 12.
  • In this manner, displaying an excavatable range B on the monitor 20 enables the driving of the hydraulic excavator 1 with reference to the excavatable range B when the hydraulic excavator 1 is travelled and revolved to select an excavation site, facilitating positional adjustment for excavation site. As a result, displaying the excavatable range B on the monitor 20 offers enhanced work efficiencies as well as the ensuring of work safety.
  • REFERENCE SIGNS LIST
    • 1 Hydraulic excavator
    • 2 Lower travel base
    • 3 Revolving upperstructure
    • 3 a Revolving upperstructure main unit
    • 4 Cabin
    • 5 Front working mechanism
    • 15, 15F, 15B, 15L, 15R Camera
    • 20 Monitor
    • 21, 21F, 21B, 21L, 21R Camera image
    • 21C Icon image

Claims (3)

1. A monitoring image display device of an industrial machine, comprising:
one or more monitoring cameras (15, 15F, 15B, 15L, 15R) installed on the industrial machine (1) having a movable mechanism provided at least in part, and placed in a location at least partially extending beyond an outline region to have a field of view covering the surroundings of the industrial machine (1);
an image generating unit (35) that generates top-view images captured by the monitoring cameras (15, 15F, 15B, 15L, 15R) looking downward from above;
an icon-image generating unit (33) that generates an icon image (21C) representing a plane of the industrial machine (1) with at least outline being shown in a graphic form;
an image compositing unit (34) that performs composite processing to display an image captured by the image generating unit (35) around the icon image (21C) generated by the icon-image generating unit (33); and
an image display unit (20) that displays the top-view images composited by the image compositing unit to include part of a section hidden within a display region of the icon image (21C).
2. The monitoring image display device of the industrial machine according to claim 1, wherein:
the industrial machine (1) has a frame (3) capable of revolving;
the icon image (21C) includes a region including the frame (3) and a projection extending outward from the frame (3), the region being displayed in an outline; and
the monitoring camera (15) is placed in a position upward of a section of the outline to jut outward from the outline.
3. The display device of the industrial machine according to claim 1, wherein:
the industrial machine (1) has a movable working mechanism (5) projecting beyond the frame (3); and
the icon image (21C) has a movable-range display region (B) for displaying a motion track of the working mechanism (5).
US14/916,058 2014-02-17 2015-01-20 Monitoring image display device of industrial machine Abandoned US20160217331A1 (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
JP2014027536A JP6095592B2 (en) 2014-02-17 2014-02-17 Monitoring image display device for hydraulic excavator
JP2014-027536 2014-02-17
PCT/JP2015/051402 WO2015122245A1 (en) 2014-02-17 2015-01-20 Monitoring image display device of industrial machine

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2015/051402 A-371-Of-International WO2015122245A1 (en) 2014-02-17 2015-01-20 Monitoring image display device of industrial machine

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/158,346 Continuation US10650251B2 (en) 2014-02-17 2018-10-12 Monitoring image display device of industrial machine

Publications (1)

Publication Number Publication Date
US20160217331A1 true US20160217331A1 (en) 2016-07-28

Family

ID=53799993

Family Applications (2)

Application Number Title Priority Date Filing Date
US14/916,058 Abandoned US20160217331A1 (en) 2014-02-17 2015-01-20 Monitoring image display device of industrial machine
US16/158,346 Active US10650251B2 (en) 2014-02-17 2018-10-12 Monitoring image display device of industrial machine

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/158,346 Active US10650251B2 (en) 2014-02-17 2018-10-12 Monitoring image display device of industrial machine

Country Status (7)

Country Link
US (2) US20160217331A1 (en)
EP (1) EP3110142B1 (en)
JP (1) JP6095592B2 (en)
CN (1) CN105493503B (en)
AU (1) AU2015216409B2 (en)
CA (1) CA2923033C (en)
WO (1) WO2015122245A1 (en)

Cited By (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20160344931A1 (en) * 2015-05-22 2016-11-24 Caterpillar Inc. Imaging system for generating a surround-view image
US20180063446A1 (en) * 2016-08-30 2018-03-01 Caterpillar Inc. Method of imaging a machine environment and machine system implementing same
US20180079361A1 (en) * 2016-09-20 2018-03-22 The University Of Tokyo Display system for work vehicle
US20180202131A1 (en) * 2015-10-23 2018-07-19 Komatsu Ltd. Display system
US20180313062A1 (en) * 2015-12-28 2018-11-01 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Shovel
US20180352162A1 (en) * 2017-06-06 2018-12-06 Caterpillar Inc. Display system for machine
US20190055708A1 (en) * 2017-08-18 2019-02-21 Independence Excavating, Inc. Slope roller assembly
US10250812B2 (en) 2017-05-17 2019-04-02 Caterpillar Inc. Display system for machine
US20190241124A1 (en) * 2016-11-01 2019-08-08 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Construction machine safety management system, management apparatus
US10590629B2 (en) * 2016-03-29 2020-03-17 Komatsu Ltd. Working vehicle
US10919584B2 (en) 2017-10-17 2021-02-16 Tadano Ltd. Work vehicle
US20210108394A1 (en) * 2018-05-25 2021-04-15 Deere & Company Object responsive control system for a work machine
US11208040B2 (en) * 2015-10-23 2021-12-28 Komatsu Ltd. Display system and work vehicle
US11231501B2 (en) 2019-09-26 2022-01-25 Baidu Usa Llc Front and side three-LIDAR design for autonomous driving vehicles
EP4033749A4 (en) * 2019-09-20 2023-10-04 Wise Automotive Corporation Front image generation device for heavy equipment
US11898332B1 (en) * 2022-08-22 2024-02-13 Caterpillar Inc. Adjusting camera bandwidth based on machine operation
WO2024149001A1 (en) * 2023-01-13 2024-07-18 北京天玛智控科技股份有限公司 Video monitoring method and apparatus for coal mine fully mechanized mining face, electronic device and medium

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2016242761B2 (en) * 2015-03-31 2019-08-08 Komatsu Ltd. Working machine
JP6713190B2 (en) * 2015-10-09 2020-06-24 住友重機械工業株式会社 Shovel operating device and shovel operating method
JP6684682B2 (en) * 2016-08-18 2020-04-22 株式会社神戸製鋼所 Construction machinery
KR20180078970A (en) * 2016-12-30 2018-07-10 화남전자 주식회사 Camera Arranging Structure for Monitoring Surrounding Area of Construction Equipment
JP6958216B2 (en) * 2017-10-17 2021-11-02 株式会社タダノ Work vehicle
JP7143578B2 (en) * 2017-10-17 2022-09-29 株式会社タダノ work vehicle
CN109255335A (en) * 2018-09-28 2019-01-22 广州希华通讯设备有限公司 Identification monitoring system for construction site excavator
WO2021054758A1 (en) * 2019-09-20 2021-03-25 주식회사 와이즈오토모티브 Front image generation device for construction equipment
WO2021054757A1 (en) * 2019-09-20 2021-03-25 주식회사 와이즈오토모티브 Front image generation device for construction equipment
CN115013386B (en) * 2022-05-30 2023-06-23 燕山大学 Hydraulic system protection device control method and control device based on visual recognition

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010030081A1 (en) * 1999-12-20 2001-10-18 Kazuyoshi Morimoto Ladder unit for climbing up and down construction machine
US20030085995A1 (en) * 2001-06-15 2003-05-08 Hiroshi Sawada Construction machine
WO2013105597A1 (en) * 2012-01-12 2013-07-18 日立建機株式会社 Periphery monitoring device for self-propelled industrial machine

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2913230B2 (en) * 1991-10-23 1999-06-28 日立建機株式会社 Lighting equipment for large construction machinery
JPH10237900A (en) * 1997-02-28 1998-09-08 Hitachi Constr Mach Co Ltd Construction machine having ladder
JP3980565B2 (en) * 2004-02-16 2007-09-26 松下電器産業株式会社 Driving assistance device
KR100966288B1 (en) * 2009-01-06 2010-06-28 주식회사 이미지넥스트 Around image generating method and apparatus
JP4951639B2 (en) 2009-03-02 2012-06-13 日立建機株式会社 Work machine with ambient monitoring device
JP5269026B2 (en) * 2010-09-29 2013-08-21 日立建機株式会社 Work machine ambient monitoring device
CN103180522B (en) * 2010-10-22 2016-01-13 日立建机株式会社 The periphery monitoring apparatus of Work machine
KR101735116B1 (en) * 2011-01-24 2017-05-15 두산인프라코어 주식회사 an Drive Information Recording Apparatus for a Construction Heavy Equipment
WO2012157379A1 (en) * 2011-05-13 2012-11-22 日立建機株式会社 Device for monitoring area around working machine
CN103635637B (en) * 2011-05-13 2015-11-25 日立建机株式会社 The back visibibility display unit of hydraulic crawler excavator
JP2012252675A (en) * 2011-06-07 2012-12-20 Komatsu Ltd Dump truck
JP5888956B2 (en) * 2011-12-13 2016-03-22 住友建機株式会社 Excavator and surrounding image display method of the excavator
JP5988683B2 (en) * 2012-05-15 2016-09-07 日立建機株式会社 Display device for self-propelled industrial machine
JP2013253402A (en) * 2012-06-06 2013-12-19 Hitachi Constr Mach Co Ltd Surrounding monitoring device for work machine
JP6014664B2 (en) * 2012-06-08 2016-10-25 日立建機株式会社 Display device for self-propelled industrial machine

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20010030081A1 (en) * 1999-12-20 2001-10-18 Kazuyoshi Morimoto Ladder unit for climbing up and down construction machine
US20030085995A1 (en) * 2001-06-15 2003-05-08 Hiroshi Sawada Construction machine
WO2013105597A1 (en) * 2012-01-12 2013-07-18 日立建機株式会社 Periphery monitoring device for self-propelled industrial machine

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Furuwatari WO 2013105597 A1, hereinafter *
Sawada US 20030085995 A1, hereinafter *

Cited By (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9871968B2 (en) * 2015-05-22 2018-01-16 Caterpillar Inc. Imaging system for generating a surround-view image
US20160344931A1 (en) * 2015-05-22 2016-11-24 Caterpillar Inc. Imaging system for generating a surround-view image
US10472806B2 (en) * 2015-10-23 2019-11-12 Komatsu Ltd. Display system
US20180202131A1 (en) * 2015-10-23 2018-07-19 Komatsu Ltd. Display system
US11208040B2 (en) * 2015-10-23 2021-12-28 Komatsu Ltd. Display system and work vehicle
US11802393B2 (en) 2015-12-28 2023-10-31 Sumitomo (S.H.I.) Construction Machinery Co., Ltd. Shovel
US20180313062A1 (en) * 2015-12-28 2018-11-01 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Shovel
US11230823B2 (en) * 2015-12-28 2022-01-25 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Shovel
US10590629B2 (en) * 2016-03-29 2020-03-17 Komatsu Ltd. Working vehicle
US20180063446A1 (en) * 2016-08-30 2018-03-01 Caterpillar Inc. Method of imaging a machine environment and machine system implementing same
US10070071B2 (en) * 2016-08-30 2018-09-04 Caterpillar Inc. Method of imaging a machine environment and machine system implementing same
US10787127B2 (en) * 2016-09-20 2020-09-29 The University Of Tokyo Display system for work vehicle
US20180079361A1 (en) * 2016-09-20 2018-03-22 The University Of Tokyo Display system for work vehicle
AU2019213435B2 (en) * 2016-09-20 2020-11-26 Komatsu Ltd. Display system for work vehicle
US11148593B2 (en) * 2016-11-01 2021-10-19 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Construction machine safety management system, management apparatus
US20190241124A1 (en) * 2016-11-01 2019-08-08 Sumitomo(S.H.I.) Construction Machinery Co., Ltd. Construction machine safety management system, management apparatus
US10250812B2 (en) 2017-05-17 2019-04-02 Caterpillar Inc. Display system for machine
US10889958B2 (en) * 2017-06-06 2021-01-12 Caterpillar Inc. Display system for machine
US20180352162A1 (en) * 2017-06-06 2018-12-06 Caterpillar Inc. Display system for machine
US20190055708A1 (en) * 2017-08-18 2019-02-21 Independence Excavating, Inc. Slope roller assembly
US10662603B2 (en) * 2017-08-18 2020-05-26 Independence Excavating, Inc. Slope roller assembly
US10919584B2 (en) 2017-10-17 2021-02-16 Tadano Ltd. Work vehicle
US20210108394A1 (en) * 2018-05-25 2021-04-15 Deere & Company Object responsive control system for a work machine
US11828046B2 (en) * 2018-05-25 2023-11-28 Deere & Company Object responsive control system for a work machine
EP4033749A4 (en) * 2019-09-20 2023-10-04 Wise Automotive Corporation Front image generation device for heavy equipment
US11231501B2 (en) 2019-09-26 2022-01-25 Baidu Usa Llc Front and side three-LIDAR design for autonomous driving vehicles
US11898332B1 (en) * 2022-08-22 2024-02-13 Caterpillar Inc. Adjusting camera bandwidth based on machine operation
US20240060276A1 (en) * 2022-08-22 2024-02-22 Caterpillar Inc. Adjusting camera bandwidth based on machine operation
WO2024149001A1 (en) * 2023-01-13 2024-07-18 北京天玛智控科技股份有限公司 Video monitoring method and apparatus for coal mine fully mechanized mining face, electronic device and medium

Also Published As

Publication number Publication date
US20190042858A1 (en) 2019-02-07
JP2015151794A (en) 2015-08-24
AU2015216409B2 (en) 2017-10-19
EP3110142A1 (en) 2016-12-28
WO2015122245A1 (en) 2015-08-20
CN105493503A (en) 2016-04-13
EP3110142B1 (en) 2021-12-22
CA2923033A1 (en) 2015-08-20
JP6095592B2 (en) 2017-03-15
CN105493503B (en) 2018-08-24
CA2923033C (en) 2021-01-12
EP3110142A4 (en) 2017-11-29
US10650251B2 (en) 2020-05-12
AU2015216409A1 (en) 2016-03-17

Similar Documents

Publication Publication Date Title
US10650251B2 (en) Monitoring image display device of industrial machine
CN107407078B (en) Periphery monitoring device for working machine and periphery monitoring method for working machine
JP7108750B2 (en) System and method
JP6204884B2 (en) Peripheral display device for swivel work machine
CN107627957B (en) Working vehicle
KR101945421B1 (en) Work-machine periphery monitoring device
EP2717570A1 (en) Device for monitoring area around working machine
JP6106754B2 (en) Work machine ambient monitoring device
JP6029941B2 (en) Image generator for paving machines
US10621743B2 (en) Processing-target image creating device, processing-target image creating method, and operation assisting system
CN111868340A (en) Construction machine
KR20130138227A (en) Perimeter surveillance device for work machine
JP2013253402A (en) Surrounding monitoring device for work machine
KR20180120665A (en) Working vehicle
CN107407075B (en) Image data generating method
KR102027395B1 (en) Display system of work machine, and display method of work machine
JP2017145649A (en) Work machine
JP2014033340A (en) Monitoring device of transport vehicle
CN108432241B (en) Excavator
US20210363732A1 (en) System and method for selectively displaying image data in a working machine
JP2015154239A (en) Bird's-eye-view image display device for construction machine
WO2020080294A1 (en) Hydraulic excavator
KR20170024295A (en) Working machine

Legal Events

Date Code Title Description
AS Assignment

Owner name: HITACHI CONSTRUCTION MACHINERY CO., LTD., JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KOWATARI, YOICHI;ISHIMOTO, HIDEFUMI;OOTA, MORITAKA;AND OTHERS;SIGNING DATES FROM 20160201 TO 20160203;REEL/FRAME:037872/0585

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION